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PREFACE 

This  thesis  is  the  result  of  a  5  month  research  that  examined  the 
application  of  a  formal  risk  assessment  methodology  suggested  by  the  Air 
Force  Academy  Risk  Analysis  Study  Team.  The  intent  was  to  determine  the 
feasibility  and  practicality  of  applying  such  a  methodology  in  an  actual 
weapons  system  acquisition  program  environment.  We  hope  that  this  paper 
will  provide  the  reader  with  some  insight  into  the  difficulties  of  apply¬ 
ing  this  risk  assessment  methodology  to  a  real  world  situation  and  perhaps 
suggest  more  appropriate  use  of  risk  assessment  in  future  research  and 
development  programs. 

We  would  like  to  thank  the  members  of  the  Directorate  of  Program 
Control  in  the  A-10  Program  Office,  principally  Captain  Bob  Cote,  and 
Mr.  H.  Stein  of  Fairchild  Republic  Company  for  their  cooperation  and 
assistance  in  this  effort. 

As  always,  the  views  and  conclusions  are  solely  our  own  and  we 
assume  full  personal  responsibility  for  errors  and  omissions. 

Stephen  L.  Amdor  and  Roy  R.  Kilgore 
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This  thesis  attempts  to  test  the  feasibility  of  a  formal  risk 
assessment  methodology  in  an  actual  weapons  acquisition  program  environ¬ 
ment.  The  methodology  essentially  consists  of  obtaining  subjective  data 
in  a  probabilistic  format  and  analyzing  it  via  a  rather  sophisticated 
network  simulation  computer  program.  If  successful,  we  anticipated  that 
the  methodology  might  well  serve  as  a  normative  base  for  risk  assess¬ 
ment  in  future  weapons  acquisition  efforts.  If  unsuccessful,  it  should 
at  least  provide  suggestions  for  alternative  approaches  to  forecasting 
contractors'  performance.  A  significant  personal  benefit  was  the  invalu¬ 
able  practical  exposure  afforded  the  writers  while  still  in  the  academic 
environment . 

Motivation 

Basic  incentive  for  the  study  arose  from  efforts  to  improve 
Department  of  Defense  (DOD)  performance  in  the  weapons  acquisition 
process.  During  the  past  decade  and  with  few  exceptions  thus  far  into 
this  one,  there  have  been  notorious  examples  of  cost  overrun,  schedule 
slippage  and  performance  degradation  within  major  weapons  system 
acquisition  efforts.  Among  the  more  recent  and  widely  publicized  are 
the  USN/Grumman  F-14  fleet  defense  interceptor,  the  USAF/Lockheed  C-5A 
heavy  logistic  aircraft,  and  the  USA/ General  Motors  main  battle  tank. 

A  plethora  of  management  approaches,  contractual  schemes,  and 
accounting  methods  has  failed  to  provide  the  desired  results.  Problems 
persist,  as  attested  to  by  the  current  difficulties  in  the  Rockwell 
International  B-l  advanced  manned  bomber  program  (51 :18).  But  there 
are  successes,  like  the  AC-130  gunehip  program  (15).  Such  successes 
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lend  credence  to  the  assertion  that  at  least  some  parts  of  current 
DOD  acquisition  policy  are  producing  positive  results.  Among  those 
aspects  which  are  seen  to  be  of  potential  benefit  is  an  increased  aware¬ 
ness  of  the  need  for  a  way  to  assess  program  risk  early  in  the  cycle, 
and  to  continue  that  assessment  as  long  as  significant  uncertainty 
remains.  The  question  then  arises,  "How  do  we  go  about  assessing 
program  risk?" 

Toward  the  end  of  improving  the  weapons  acquisition  process,  former 
Deputy  Secretary  of  Defense  David  Packard  issued  two  memoranda  on  the 
subject  to  the  Service  Secretaries.  In  chose,  Secretary  Packard  directed 
the  Secretaries  to  identify  areas  of  high  technical  risk,  to  accomplish 
"forma L  risk  analysis"  and  to  expand  program  management  practices  to 
include  explicit  consideration  of  risk  assessment,  risk  reduction,  and 
risk  avoidance  (19 :1) .  In  addition,  Department  of  Defense  Directive 
(DODD)  5000.1  also  contains  reference  to  risk,  requiring  that  the 
Development  Concept  Paper  (DCP)  define  program  issues,  including  risk, 
and  that  technical  uncertainty  be  continually  assessed  (21) .  Also, 

Air  Force  Systems  Command  (AFSC)  Pamphlet  800-3  addresses  risk  analysis 
in  conjunction  with  the  Defense  Systems  Acquisition  and  Review  (DSARC) 
process  (3) . 

In  response  to  these  general  guidelines,  the  Aeronautical  Systems 
Division  (ASD)  of  Air  Force  Systems  Command  commissioned  a  study  by 
members  of  the  faculty  of  the  United  States  Air  Force  Academy  (USAFA) 
on  risk  analysis  (33) .  We  feel  that  the  USAFA  study  is  the  collative 
document  on  risk  analysis  within  DOD.  One  of  the  observations  in  that 
report  serves  as  pointed  motivation  for  this  thesis: 

The  technique  which  offers  the  most  promise  in  quantitative 

risk  assessment  is  a  versatile,  simulated  network  approach  using 
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group  assessment  techniques,  subjective  probability,  technological 
forcasting,  coat  estimating,  anil  other  sources  of  input  (33:73) . 

Further, 

Conclusion:  To  our  knowledge  no  major  DOD  program  has  developed  or 
used  a  risk  analysis  of  the  magnitude  envisioned  in  this  report. 

Recommendation!  Initiate  test  cases  immediately.  Formal  risk 
assessment  and  analysis  should  be  used  throughout  these  pilot 
programs  to  determine  their  feasibility  and  utility  to  a  daciaion 
maker  (33:9). 

Background 

Subsequent  to  the  publication  of  that  study  at  least  one  attempt 
has  been  made  to  test  the  feasibility  of  a  simulated  network  approech 
to  risk  assessment.  Bevelhymer  wrote  a  thesla  baaed  on  hla  efforts  to 
test  such  an  approach  using  the  AGM-86A  (SCAD)  program  aa  a  teat  caae 
(10). 

The  principal  difference  between  Bevelhymer 's  work  and  that 
proposed  in  this  thesis  lies  in  the  acquisition  and  treatment  of  data. 
Bevelhymer  noted,  as  we  do,  that  the  preponderance  of  risk  analysis 
literature  suggests  Che  use  of  subjectively  obtained  data,  but  his 
questions  concerning  hov;  the  "experts"  should  be  chosen  and  how  they 
should  be  questioned  led  him  to  initially  attempt  the  simulation 
with  data  drawn  from  existing  contract  data  documents  (10:26). 

Allowing  the  major  data  premise  in  risk  analysis,  that  is,  that 
cost,  time  and  performance  parameters  are  realizations  of  random 
variables,  one  must  obtain  a  range  of  estimates  for  each  parameter 
considered.  Bevelhymer  could  not  get  such  information  solely  from 
contractor  supplied  data  items,  which  are  deterministic  in  nature. 

He  therefore  was  forced  to  resort  to  subjective  estimates  for  range 
endpoints  used  to  normalize  the  contractor  supplied  cost  observations 
(10:37). 
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We  felt  that  this  compromise,  for  the  sake  of  methodological, 
coat  reduction,  vaa  questionable,  aspects) ly  in  light  of  the  feet  that 
ooat  endpoints  were  ultimately  supplied  by  an  independent  firm  under 
oontraot  to  the  SCAD  System  Program  Office  (8F0)  to  aaaiat  in  ayatema 
integration  and  technological  assessment.  That  was  an  atypical  situation 
certainly  not  without  real  coat  <10i 3?) .  Further,  Bevelhymer  had  great 
difficulty  with  time  estimates  because  no  contract  data  elements  exrlirit- 
ly  provided  f&y  time  estimate,  much  leas  ranges  of  estimates.  Since  he 
could  not  use  contract  time  data  items,  he  vaa  forced  to  search  elsewhere 
and  eventually  obtained  subjective  estimates  from  the  Projects  Division 
within  the  SCAD  SPO  (10»37). 

We  therefore  chose  to  attempt  a  subjective  assessment  approach  to 
data  acquisition.  Initially,  we  Intended  to  use  both  the  SPO  and 
contractor  personnel  in  two  separate  iterations  to  afford  a  comparative 
baae.  Tha  SPO  effort  proved  unfeasible  simply  because  the  detailed  level 
of  expertise  did  not  exist  there.  Consequently  we  attempted  to  uee 
"experts"  from  among  the  contractor1 i  personnel,  an  intuitively  optimum 
choice  if  one  either  assumes  lack  of  bias  or  corrects  for  its  presance. 

Va  attempted  to  do  tha  latter,  as  described  in  Chapter  4. 

Although  Bevelhymer' s  work  ia  the  only  actual  teat  case  using  net¬ 
work  analysis  that  w«  found,  others  have  advocated  it  uaed  and  even 
produced  hypothetical  examples.  See,  for  instance,  Williams  (S3),  Brandt 
(13),  Hwang  <31 ),  and  Moeller  (37).  While  not  a  network  based  technique, 
Thomas  has  successfully  employed  an  accumulative  approach  using  subjec¬ 
tive  inputs  and  producing  probabilistic  estimates  of  total  risk  in  biddor 
replies  to  a  Request  for  Proposals  (49). 
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Overview 

This  thesis  is  organised  into  seven  chapters  and  associated 
appendices.  Problem  statement  and  background  are  in  this  chapter* 

Chapter  Two  discusses  risk  assessment  syntax  and  the  assumptions 
pertinent  to  the  methodology.  In  Chapter  Three  a  brief  diacusaion  of 
the  current  Department  of  Defense  management  system  is  presented »  along 
with  some  of  its  shortcomings.  The  proposed  methodology  ie  presented 
in  Chapter  Four.  In  Chapter  Five  a  comprehensive  assessment  of  the 
fellure  of  the  methodology  is  given  and  the  problem  is  restated  in 
terms  of  the  program  manager's  requirements.  Chapter  Six  briefly 
discusses  some  secondary  attempts  at  forecasting  contractor  performance 
using  contract  data  alemanta  and  lsaat  squares  analyels.  Chapter  Seven 
contains  our  conclualono  end  recommendetiona  for  further  study. 

To  recapitulate,  this  theeis  attempts  to  test  the  feasibility  of  s 
formal  risk  assessment  methodology  that  uses  subjectively  based  probabil¬ 
istic  estimates  as  inputs  for  s  computerised  network  simulation.  The 
prime  criterion  for  judging  the  feasibility  is  whether  or  not  the 
methodology  provides  useful  information  to  the  decision  maker  under 
reasonable  constraints  like:  (s)  is  the  information  produced  cogent, 

(b)  is  it  accurate,  and  (c)  is  it  timely. 
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XX.  Syntax  and  Assumptions 

Thin  chapter  establishes  a  riak  aaaaaamont  syntax,  explains 
pertinent  concepts,  and  lists  assumptions  used  in  the  study. 

Uncertainty 

Among  the  adjectives  that  might  be  used  to  describe  the  weapons 
system  acquisition  environment  is  one  that  seems  particularly  appropriate 
uncertain.  The  implication  of  "uncertainty"  ranges  from  lack  of  absolute 
sureness,  to  lack  of  conviction,  to  no  opinion,  to  confusion,  and  so 
forth.  We  again  draw  upon  the  USAFA  study  on  risk  analysis  for  syntac¬ 
tical  convention: 

Uncertainty:  Incomplete  knowledge  (33*8). 

Further,  most  works  on  risk  analysis  are  primarily  directed  toward 
recognition  and  assessment  of  technical  uncertainties  (13,  3,  32). 

The  USAFA  study  categorises  uncertainty  in  the  weapons  system  acquisition 

i  « 

process  in  four  interrelated  areas. 

Target  uncertainty  is  the  uncertainty  associated  with  defining 
a  need  or  required  operational  capability  (ROC)  and  reducing  that  need 
to  cost,  schedule,  and  performance  goals  (33:23-25).  Principal  among 
the  factors  contributing  to  target  uncertainty  are:  (a)  validity  of  the 
need,  (b)  confusion  resulting  from  the  formal  requirements  generation 
process,  (c)  questions  concerned 'with  the  physical  and  performance 
characteristics  required  to  meet  the  need,  and  (d)  inherent  innaccuracics 
associated  with  time  and  cost  estimating  techniques.  Target  uncertainty 
stems  from  the  lack  of  answers  to  the  question  "What  do  we  need?" 

Technical  uncertainty  is  closely  related  to  target  uncertainty, 
the  essential  difference  being  that  technical  uncertainty  addresses 
the  question  "  Can  it  be  done  at  all,  for  any  price?"  (33:25-28). 
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The  distinction  between  uncertainly  of  the  criteria  used  in  solving  the 
problem  and  the  technical  solution^  itself  must  be  made.  The  answers  to 
"Does  the  technical  solution  lie  beVond  program  time  and  cost  constraints? 
and  to  "Is  the  solution  beyond  present  technological  capabilities?" 

Impinge  upon  the  degree  of  technological  uncertainty.  Unanticipated 

\ 

technological  problems  add  even  more  uncertainty  to  the  estimation  process 
These  technological  problems  have  been  prevalent  in  recent  years,  largely 
due  to  the  "doctrine  of  quality"  philosophy  among  U.S.  weapons  designers. 
This  philosophy  prefers  advanced  technology  often  at  the  expense  of 
quantity  (20 si).  This  attitude  has  only  recently  been  subjected  to 
question  and  the  test  case  for  this  study,  the  Fairchild  Republic  A-10 
specialized  close  air  support  aircraft,  is  one  result.  The  phrase  applied 
to  this  approach  is  "design  to  cost"  (1:22). 

Internal  program  uncertainty  is  associated  with  selecting  a 
particular  management  approach  to  the  problem  and  then  carrying  it  out. 
(33:28-30).  The  results  of  a  particular  management  style  or  philosophy 
are  impacted  by  uncertainties  related  to  estimates  in  the  target  and 
technical  areas,  by  the  particular  acquisition  strategy  selected,  and 
are  impacted  by  arguments  among  Congress,  force  structure  planners,  threat 
assessors,  and  users.  These  vasclllatlons  among  interested  agencies 
are  part  of  the  final  category  of  uncertainty,  process  uncertainty. 

Process  uncertainty  pervades  the  other  three  categories,  as  mention¬ 
ed  above.  The  process  uncertainties  derive  from  influences  external  to 
the  program  itself  (33:30-33).  in  addition  to  those  areas  listed  as 
affecting  internal  program  uncertainty,  the  process  area  includes 
Interservice  rivalry,  national  policy,  budgetary  considerations,  and 
Congressional  "pork  barrel"  activity.  Obviously,  this  area  is  the  one 
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the  program  manager  has  the  least  control  over.  Process  uncertainty 
may  prevail  over  all  attempts  at  assessing  and  controlling  the  other 
three  types. 

Formal  Risk  Analysis 

Lack  of  explicit  direction  on  how  to  accomplish  a  "risk  analysis" 
has  led  to  several  approaches,  some  qualitative,  the  B-l  bomber  program 
for  Instance  (39) »  and  some  quantitative,  such  as  Thomas'  method  (49). 

For  the  purposes  of  this  study  we  consider  "formal"  to  mean  a  separate 
and  documented  effort,  conducted  in  accordance  with  normally  accepted 
scientific  investigative  criteria.  From  the  USAFA  stu*y  come  other 
pertinent  definitions: 

Risk;  The  probability  that  a  planned  event  will  not  be  attained 
within  constraints  (cost,  schedule,  performance)  by  following  a 
specified  course  of  action,  {note  that  risk  is  the  complement  of 
the  cumulative  probability,  i.e.,  R«l-Pr(X  less  than  or  equal  to  x)]. 

Risk  assessment:  A  comprehensive  and  structured  process  for  estimating 
the  risk  associated  with  a  particular  alternative  course  of  action; 
also  the  product  of  such  a  process. 

Risk  management:  The  generation  of  alternative  courses  of  action 
for  reducing  risk.  [Sometimes  called  risk  avoidance]. 

Risk  analysis:  The  process  of  combining  risk  assessment  with  risk 
management  in  an  iterative  cycle;  also  the  produce  of  such  a  process 
(33). 

Parameters 

As  noted  previously,  three  parameters,  or  three  classes  of  para¬ 
meters,  are  used  to  quantify  these  uncertainties.  They  are  cost,  time 
and  performance.  By  cost,  we  mean  dollar  costs  for  obtaining  the  hard¬ 
ware,  software,  and  services  necessary  to  fulfill  the  contractual 
obligations.  Time  and  scope  limitations  on  this  thesis  preclude 
addressing  such  things  as  opportunity  costs,  depreciation,  and  the  very 
real  costs  attributable  to  federal  administration  of  the  program. 
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However,  increased  emphasis  is  being  placed  on  those  excluded  costs  and 
there  is  certainly  room  for  further  study  (see  Ref.  2). 

The  time  parameter  is  calendar  time  from  contract  award  and  is 
associated  with  periods  between  significant  contractual  milestones,  e.g., 
rollout,  first  flight,  etc.  Certain  government  specified  milestones 
also  become  significant  but  are  not  necessarily  hardware  oriented.  The 
Defense  System  Acquisition  and  Review  Council  meetings  are  examples  (3). 

The  performance  parameter  is  the  most  ill-defined  and  perplexing. 

The  many  factors  that  could  be  used  in  any  one  program  depend  on  the 
weapons,  the  threat,  the  mission,  and,  not  insignificantly,  each  other. 
Performance  parameters  are  not  homogeneous  and  hence  don't  readily  lend 
themselves  to  an  additive  scheme  like  network  simulation.  For  example, 
the  Development  Concept  Paper  for  the  A- 10  (18)  specified  such  general 
performance  requirements  as  high  payload  capacity,  small  turn  radius, 
and  long  loiter  time.  As  yet,  there  are  no  transformations  that  allow 
additive  accumulation  <of  some  performance  parameter  that  simultaneously 
represents  ordnance  capacity,  radial  acceleration,  and  tlme-on-target 
achievement  levels. 

Assumptions 

As  noted  earlier,  these  parameters  must  be  considered  as  random 
variables  if  a  probabilistic  estimate  of  program  success  (in  one  or  more 
of  the  parameters)  is  to  be  produced.  This  condition  leads  to  the  first 
of  several  assumptions  used  in  this  thesis,  namely  that  subjectively 
estimated  distributions  of  each  parameter  are  marginal  distributions 
(27).  This  assumption  does  not  necessarily  imply  Independence  among 
the  parameters.  Peck  and  Scherer  point  out  that  observat  ms  by  program 
managers  tend  to  support  the  hypothesis  that  there  is  Indeed  some 
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dependence,  although  it  has  not  been  proven  empirically  (40).  If 
independence  is  not  so,  the  conclusion  must  be  that  there  exists  some 
trivariate  distribution  for  each  case,  although  its  form  is  unknown. 

That  is,  there  is  some  unknown  relationship  among  the  three  variables 
that,  were  it  known,  would  allow  the  manager  to  make  tradeoffs  among 
the  three  under  a  specified  total  risk  constraint,  R,  where  R  lies 
in  the  range  0  to  1.0. 

Further,  we  chose  to  disregard  the  performance  aspect  of  the  risk 
assessment  and  to  concentrate  on  only  the  time  and  cost  parameters. 

Three  considerations  affect  this  assumption.  First,  as  noted  earlier, 
the  performance  parameter  is  not  homogeneous  and  hence  is  not  tractable 
in  a  network  simulation.  Second,  the  nature  of  this  particular  test 
case,  the  A- 10  specialized  close  air  support  aircraft,  reduces  the 
potential  impact  of  not  considering  the  performance  variable.  The 
A-10  is  a  low  state-of-the-art  venture  not  requiring  significant 
technical  advances.  Further  technical  risk  reduction  came  from  the 
Competitive  Prototype  Phase  (CPP)  wherein  two  prototypes  were  built 
and  tested  in  an  operational  setting.  For  further  information  on  the  A-10 
program,  see  Appendix  A.  Thirdly,  time  available  for  the  study  precluded 
attempting  all  three.  We  acknowledge  the  questionability  of  this 
assumption  in  most  cases  since  virtually  all  of  the  literature  and 
most  of  the  operators  we  have  interviewed  look  upon  risk  analysis 
as  technical  risk  analysis  (53). 

A  third  major  assumption  was  that  we  would  be  able  to  construct 
a  valid  summary  network,  with  well  defined  activities  or  groups  of 
activities,  based  on  the  Work  Breakdown  Structure  (WBS).  The  WBS  Is 
an  Integral  part  of  the  DOD  management  system  and  will  be  explained  In 
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some  detail  in  Chapter  3.  For  now,  the  definition  take  from  Military 
Standard  881  will  suffice: 

Work  breakdown  structure  (WBS) .  A  work  breakdown  structure  is  a 
product-oriented  family  tree  composed  of  hardware,  software,  serv¬ 
ices,  and  other  work  tasks  which  result  from  project  engineering 
efforts  during  the  development  and  production  of  a  defense  materiel 
item,  and  which  completely  defines  the  project/program.  A  WBS 
displays  and  defines  the  product (s)  to  be  developed  or  produced 
and  relates  the  elements  to  be  accomplished  to  each  other  and  to 
the  end  product  (35:2). 

A  fourth  assumption  was  that  among  the  contractor  personnel  we  could 
find  experts  who  could  give  us  time  and  cost  estimates,  in  the  form  of 
most  likely,  minimum,  and  maximum  points,  for  each  activity  in  the  WBS 
derived  network.  As  a  corollary,  we  assumed  that  the  estimates  so  rend¬ 
ered  would  be  sufficiently  accurate  and  unbiased  as  to  afford  a  measure 
of  reliablility  in  the  ultimate  answer.  This  assumption,  when  constrained 
by  the  summary  network  criterion  mentioned  in  assumption  three,  proved 
to  be  incorrect. 

The  fifth  assumption  was  that  we  need  not  concern  ourselves  with 

the  entire  program  but  could  limit  the  study  to  the  airframe  alone, 

Including  the  gun  and  engine  as  unit  activities  only.  The  assumption 

was  ultimately  altered  by  adverse  developments  in  the  study  so  that 

we  eventually  restricted  our  attention  to  just  a  portion  of  the  Fairchild 

Republic  Company  (FRC)  effort,  namely  the  Basic  Structure  element.  The 

FRC  task  description  of  the  Basic  Structure  element  1st 

The  design,  development,  fabrication,  assembly,  procurement  of 
parts  and  materials,  inspection,  installation,  and  functional 
test/checkout  of  the  structural,  aerodynamic,  subsystems  components 
of  the  airframe,  and  the  final  assembly  and  functional  checkout  of 
the  air  vehicle.  Includes  the  fuselage,  nacelles,  empennage,  wing 
wing  control  surfaces,  pylons,  and  assembly  thereof  (16:2-18) . 

For  further  information  on  the  A-10  Full  Scale  Development  program 
the  reader  is  again  referred  to  Appendix  A. 
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III.  The  Current  System 

As  a  prerequisite  to  understanding  the  proposed  methodology,  one 
must  first  be  familiar  with  certain  aspects  of  the  Department  of  Defense 
Resource  Management  System  (RMS),  namely  a  subsystem  thereof  called 
Selected  Acquisitions  Information  and  Management  System  (SAIM3)  (24;10). 
This  chapter  presents  a  simplified  review  of  pertinent  aspects  of  SAIMS 
and  of  the  Work  Breakdown  Structure,  which  is  the  reporting  foundation 
for  SAIMS.  The  reader  familiar  with  SAIMS  and  WBS  terms  can  safely  skip 
to  the  section  on  problems  and  shortcomings  on  Page  22. 

SAIMS  Review 

As  Inferred  above,  SAIMS  is  but  one  of  four  subsystems  (sva  Figure 
3-1)  of  the  DOD  Resource  Management  system.  SAIMS  itself  is  fur  titer  sub¬ 
divided  into  three  sections.  These  relationships  are  depicted  in  Figure 
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Figure  3-1  RMS  -  SAIMS  Relationships 
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Performance  Measurement.  We  are  concerned  with  only  one  of  the 
three  subsections  of  SAIMS,  namely  Performance  Measurement.  As  a  note 
of  caution,  the  word  "performance’1  takes  on  two  different  meanings 
throughout  RMS  literature  and  so  might  prove  confusing.  In  the  phrase 
"performance  measurement"  it  means  an  assessment  of  the  contractor's 
progress  on  all  aspects  of  the  contract.  On  the  other  hand,  in  "tech¬ 
nical  performance"  it  implies  some  measurement  of  hardware/software 
capability,  e.g.,  airspeed,  payload,  etc.  It  is  also  important  to 
realize  at  the  outset  that  each  report  derived  through  SAIMS  ultimately 
comes  from  information  taken  from  the  contractor's  data  base  and  as  such 
constitutes  a  real  cost  to  the  government. 

In  any  attempt  to  assess  the  effectiveness  of  a  risk  assessment 
methodology  designed  to  track  performance  to  date  and  predict  the 
future  behavior  of  the  three  contract  variables,  one  must  consider  what 
Is  presently  being  done  to  analyze  and  control  those  variables.  Hie 
Performance  Measurement  subsections  of  SAIMS  delineates  that  effort. 
There  are  three  basic  elements  of  the  Performance  Measurement  subsystem, 
only  two  of  which  are  important  here,  the  Cost/Schedule  Control  System 
Criteria  and  the  Cost  Performance  Report. 

The  Cost/Schedule  Control  System  Criteria  (C/SCSC)  are  used  to 
evaluate  the  effectiveness  of  the  contractor's  internal  systems. 

The  C/SCSC  do  not  require  any  data  to  be  reported  to  the  Govern¬ 
ment,  but  do  provide  for  access  to  data  needed  to  evaluate  the 
system  and  monitor  its  operation  during  the  life  of  the  contract. 
[The  Air  Force  equivalent  of  C/SCSC  is  Cost/Schedule  Planning  and 
Control  Specification,  C/SPCS  J 

The  Cost  Performance  Report  (CPR)  is  the  vehicle  which  actually 
provides  cost  and  schedule  Information  to  the  DOD  project  office. 
The  CPR  is  a  monthly  report  of  contractual  progress  with  Identi¬ 
fication  of  significant  problems  obtained  through  analyses  of 
variances  from  plans  (24s47). 

Work  Breakdown  Structure.  To  facilitate  orderly  aggregation  of 
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information  under  C/SCSC  for  the  Cost  Performance  Report,  the  SAIMS 
system  uses  the  Work  Breakdown  Structure  concept  previously  mentioned. 
Repeating,  a  WBS  is  y  product  oriented  family  tree  structure  of  all  the 
hardware,  software,  services  and  other  tasks  of  the  system  to  be  devel¬ 
oped,  produced,  supported,  and/or  operated.  Depending  on  the  user,  there 
are  actually  four  types  of  Work  Breakdown  Structures.  We  are  concerned 
with  a  portion  of  the  Contract  Work  Breakdown  Structure.  Those  defini¬ 
tions  and  others  from  Military  Standard  881,  "Work  Breakdown  Structures 
for  Defense  Materiel  Items"  are  listed  here  for  information  and  clarity. 

Summary  work  breakdown  structure  (Summary  WBS).  A  summary  work 
breakdown  structure  consists  of  the  upper  three  levels  of  a  WBS 
prescribed  by  this  standard  and  having  uniform  element  terminology, 
definition,  and  placement  in  the  family-tree  structure.  The  upper 
three  levels  of  a  summary  WBS  have  been  organized  within  the  follow¬ 
ing  categories  of  defense  materiel  items : 

a.  Aircraft  system 

b.  Electronics  system 

c.  Missile  system 

d.  Ordnance  system 

e.  Ship  system 

f.  Space  system 

g.  Surface  vehicle  system 

Level  Identification.  The  three  levels  specified  are  defined  as 
follows': 

Level  1.  Level  1  is  the  entire  defense  materiel  item;  for 
example,  the  Minute-man  ICBM  System,  the  LHA  Ship  System,  or  the 
SK-138  Self-Propelled  Howitzer  System.  Level  1  is  usually  directly 
identified  in  the  DOD  programwing/budget  system  either  as  an  integral 
program  element  or  is  a  project  within  an  aggregated  program  element. 

Level  2.  Level  2  elements  ere  major  elements  of  the  defense 
materiel  Item;  for  example,  a  ship,  an  air  vehicle,  a  tracked 
vehicle,  or  aggregations  of  services,  data,  and  activities;  for 
example,  systems  test  and  evaluation. 

Level  3  Level  3  elements  are  elements  subordinate  to  level 
2  major  elements;  for  example,  an  electric  plant,  an  airframe,  the 
power  package/drive  train,  or  type  of  service;  for  example,  techni¬ 
cal  evaluation.  (Balov  level  3,  the  contractor  identifies  the  levels] 
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Project  summary  work  breakdown  structure  (Project  summary  WBS).  A 
project  summary  work  breakdown  structure  is  a  summary  WBS  tailored 
to  a  specific  defense  materiel  item  by  selecting  applicable  elements 
from  one  or  more  summary  WBS(s)  or  by  adding  equivalent  elements 
unique  to  the  project. 

Contract  work  breakdown  structure  (Contract  WBS).  Contract  work 
breakdown  structure  is  defined  as  the  complete  WBS  for  a  contract 
developed  by  a  contractor  in  accordance  with  this  standard  and  the 
contract  work  statement.  The  contract  WBS  comprises  the  selected 
project  summary  WBS  elements  included  in  the  contract  and  those 
extensions  by  the  contractor  which  cover  the  lower  levels  of  WBS. 

Project  work  breakdown  structure  (project  WBS).  Project  work  break¬ 
down  structure  is  defined  as  the  complete  WBS  for  the  project, 
containing  all  WBS  elements  related  to  the  development  and/or 
production  of  the  defense  materiel  item.  The  project  WBS  evolves 
from  the  project  summary  WBS  extended  to  include  all  contract  WBS(s) 
and  equivalent  WBS(s)  resulting  from  DOD  in-house  efforts. 

Work  breakdown  structure  element.  A  work  breakdown  structure  element 
is  a  discrete  portion  of  a  work  breakdown  structure.  WBS  elements 
may  be  either  an  identifiable  product,  set  of  data,  or  a  service. 
(35:2-3). 

As  noted,  we  were  ultimately  concerned  in  this  study  with  a  specific 
element  of  a  Contract  WBS,  namely  a  level  4  element  called  Basic  Structure 
from  the  Fairchild  Republic  Company  (FRC)  Contract  WBS  for  the  A-10  Full 
Scale  Development  Program  (commonly  called  the  DT&E  phase,  for  Develop¬ 
ment,  Test  and  Evaluation).  The  pertinent  portion  of  the  FRC  Contract 
WBS  is  reproduced  in  Figure  3-2  as  an  aid  to  understanding  the  WBS. 
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Figure  3-2  Basic  Structuro  WBS 
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Work  Packages.  Each  level  6  WBS  element  (in  this  case)  is  further 
subdivided  into  a  set  of  work  packages.  A  work  package  is  a  short  span 
discrete  task  identified  by  the  contractor  as  part  of  the  WBS  element. 

The  work  package  is  the  fundamental  building  block  of  the  WBS  and  is 
also  the  basic  control  point  for  managing  and  accounting.  A  work 
package  has  the  following  characteristics:  (a)  it  is  a  unit  of  work  at 
the  working  level,  (b)  it  is  unique,  (c)  it  is  assigned  to  a  single 
operating  organization,  (d)  it  has  scheduled  start  and  completion  mile¬ 
stones,  (e)  it  has  a  budget  in  dollars,  hours  or  other  units,  (f)  it 
is  of  small  size  and  short  duration  to  minimize  effort  involved  with 
assessing  work- in-process  levels,  (g)  it  is  integrable  both  with  its 
cost  account  and  with  the  appropriate  WBS  element,  and  (h)  it  has  a 
specific  method  for  planning  and  measuring  performance  (17:34). 

There  are  three  types  of  work  packages  used  in  the  FRC  Management 
Control  and  Information  System.  The  unfamiliar  reader  would  do  well  to 
remember  them  and  the  methods  which  each  uses  to  measure  performance 
because  the  subject  arises  again  later  in  this  chapter  and  in  Chapter 
4.  A  Discrete  work  package  is  one  that  has  a  specific,  tangible  and 
measurable  result  or  output.  A  Level  of  Effort  (LOE)  work  package 
contains  work  that  can  be  planned  and  controlled  by  time  phased  budgets, 
l.e.,  work  that  cannot  be  identified  into  Discrete  work  packages.  An 
Apportioned  Effort  work  package  contains  work  that  is  directly  related 
to  Discrete  work  packages  of  another  cost  account  and  is  not  readily 
divisible  into  short  span  work  packages.  The  ratio  (%)  of  the  Apportioned 
Effort  to  its  related  Discrete  work  package  must  be  specified.  0-7  : 33) 

Cost  Accounts.  Sets  of  like  work  packages  conforming  to  a  part  of 
a  WBS  Control  Level  Element  (CLE,  e.g.,  the  6  level  element  1.1. 2. 1.9 
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FUSELAGE  INTEGRATION  AND  ASSEMBLY  shown  la  Figure  3-2)  ere  functionally 
grouped  into  tasks  of  work  directly  assignable  to  specific  program 
functional  organisations  like  Design,  Tool  Planning,  Lofting,  etc.  Theae 
assignments  produce  Coot  Accounts.  Cost  accounts  identify  every  program 
functional  organisation  with  work  to  do  on  a  Control  Level  Element,  and 
therefore  a  discrete  set  of  (cross  functional)  cost  accounts  will  sum  to 
a  single  WBS/CLE.  Types  of  cost  accounts  conform  to  the  types  or  work 
packages  that  make  them  up,  i.e.,  Discrete,  LOE,  and  Apportioned  Effort. 
With  the  exception  of  allowing  10%  LOE  work  packages  in  a  Discrete  cost 
account,  the  makeup  is  homogeneous.  That  is,  there  cannot  be  a  mix  of 
types  of  work  packages  in  a  given  type  of  cost  account. 

As  cautioned  earlier,  the  method  of  work  package  performance  measure¬ 
ment  is  Important.  Since  openings  and  closings  of  work  packages  cannot 
readily  be  made  to  conform  to  the  monthly  CPR  reporting  period,  some 
estimate  of  progress,  for  CPR  purposes,  must  be  made  while  the  package 
is  open.  In  the  case. of  FRC  work  packages  performance  can  be  meaoured 
four  ways:  (a)  standards  like  accepted  time  and  motion  tables,  etc.,  (b) 
50%  start  -  50%  completion,  (c)  milestones,  and  (d)  passage  of  time  (for 
LOE  packages  only) .  In  the  50%  start  -  50%  completion  case  a  package 
that  is  actually  99%  complete  at  CPR  accounting  time  will  only  be  shown 
to  be  50%  complete.  Similarly,  a  package  that  has  100%  of  its  "value" 
assigned  to  a  closing  milestone  may  be  quite  close  to  completion  yet  show 
no  progess  at  all  on  that  particular  CPR.  The  assumption  is,  of  course, 
that  the  combination  of  short  span  and  over-assignment  of  other  work 
packages  will  work  to  correct  the  mis-information  on  the  next  CPR.  It 

can  be  inferred  from  the  above  discussion  that  a  similar  thing  occurs  on 

I 

a  more  grand  scale  at  the  cost  account  level. 
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Figure  3-3  help*  clarify  the  Kooaaa.  Again,  a  Control  level 
Element  would  be  one  of  the  6  level  elemente  Hated  on  Figure  3-2.  Let 
WB8  elemente  bo  numbered  on  Figure  3-3  and  funotional  area*  be  lettered. 

In  coat  account  03  on  the  diagram  there  are  six  work  paokagea,  two  in  each 
of  three  teak  organisation*  in  the  G  functional  area.  Horiaontally  G3 
contribute*  to  the  aum  making  up  CLE  number  3.  The  aggregation  pvooe*a 
continue*  upward  for  management  information  and  to  the  left  for  CPR 
information. 

Analytic.  To  analyte  the  effect iveneae  of  the  contractor* a  effort*, 
data  la  accumulated  at  tho  coat  account  level  and  eummod  through  the  WB8 
at-ructute  to  level  3,  whore  they  are  reported  in  the  CPR.  Tho  C/8CS 
Criteria  roqulre  thet  the  contractor  be  able  to  provide  the  following 
information : 

1.  Budgeted  Cost  for  Work  Performed  (BCWP).  The  eum  of  the  budgets 
for  coup 1 e t o d  work  package a  and  completed  portions  of  open  work  pack- 
agee,  plus  budgets  for  level  of  effort  (LOR)  end  apportioned  effort 
activity  completed.  (What  you  planned  to  spend  for  the  work  you 
actually  accomplished  -  earned  value,  EV  ] 

2.  Budgeted  Cost  for  Work  Scheduled  (BCWS).  The  sum  of  the  budgets 
for  work  packages  scheduled  to  be  accomplished  (including  in-process 
work  packages),  plus  the  amount  of  leva!  of  effort  and  apportioned 
effort  scheduled  to  be  accomplished  within  a  given  period  of  time. 
[What  you  plan  to  spend  for  the  work  you  plan  to  accomplish  -  planned 
value,  PV  1 

3.  Actual  Cost  of  Work  Performed  (ACWP).  The  sum  of  actual  costs 
for  completed  work  packages  and  completed  portions  of  open  work 
packages,  plus  costs  associated  with  level  of  effort  (LOE)  and 
apportioned  effort  activity  completed.  [Dollar  value  of  resources 
consumed  in  the  accomplishment  of  work  performed-actual  costs,  AC] 

A*  Budgeted  Cost  at  Completion  (BCG).  The  program  budget. 

3.  Estimated  Cost  to  Completion  (ETC).  How  much  it  will  coot  to 
complete  the  program. 

6.  Estimated  Cost  at  Completion  (EAC).  The  sum  of  ACWP  and  ETC. 

7.  Cost  and  Schedule  Varainces  with  explanations. 

8.  Traceability.  How  and  where  you  got  the  information  (24 * 50 » 
4:121). 


18 


GSA/SM/7'I  1 


Itamo  1  through  7  are  provided  in  the  CPR.  The  format  for  the 
CPR  ia  epecified  in  the  DOD  Data  Item  Deacription  DI-F-  6000A. 

The  concept  of  "earned  value"  or  BCWP  is  important  to  the  analysis. 
Without  it,  ambiguity  arises.  For  instance  if,  in  Figure  3-4,  only  the 
solid  (actual  cost)  and  long  dashed  (bildget)  lines  were  available  one 
would  not  be  sure  whether  the  program  was  ahead  of  or  behind  schedule. 

The  apparent  coat  and  schedule  variances  are  shown  as  "Cost  Overrun  ($)" 
and  as  "Ahead  of  Schedule"  or  "Behind  Schedule"  'on  the  left  side  of 
the  "time  now"  line.  Does  the  actual  cost  point  imply  that  the  work 
scheduled  has  been  accomplished  sooner  than  expected  for  the  budgeted 
cost  or  does  it  perhaps  mean  that  the  contractor  has  spent  "Cost  Over¬ 
run  ($)"  dollars  too  much  to  achieve  the  budgeted  work,  or  is  it  a 
combination  of  the  two?  Addition  of  the  short  dashed  line  (earned  value) 
clarifies  the  picture.  The  dollar  value  of  work  associated  with  the 
Intersection  of ‘the  short  dashed  line  and  the  "time  now"  line  has  been 
"earned."  The  right  .side  of  the  figure  correctly  depicts  what  has 
actually  happened.  By  comparing  earned  value  line  to  budget  line  or  actual 
line,  we  can  measure  performance  against  actual  achievement.  It  is  clear 
that  the  schedule  variance  is  adverse,  that  is,  we  have  not  accomplished 
what  we  planned  to  at  the  time  we  planned  to  do  it.  Cost  variance  is 
also  seen  to  be  worse  than  shown  on  the  left.  Further,  by  adding  an 
Estimate  to  Completion  (ETC)  to  the  actual  cost  figure,  we  can  arrive 
at  an  Estimate  at  Completion  (EAC) .  From  that  we  subtract  planned  value 
at  completion  and  arrive  at  an  estimated  cost  variance  at  completion. 

To  summarize  the  procedure,  we  list  these  simple  formulas: 

1.  Earned  value  -  planned  value  ■  schedule  variance  (in  dollars) 

2.  Earned  value  -  actual  cost  “*  cost  variance  (also  in  dollars) 


EARNED  VALUE  CONCEPT 
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Figure  3-4.  Earned  Value  Concept 
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3.  Actual  cost  +  estimate  to  completion  =  estimate  at  completion. 
Should  the  reader  still  find  the  concepts  unclear,  a  more  detailed 
discussion  of  Performance  Measurement  within  SAIMS  is  given  in  Reference 
24. 

Problems  and  Shortcomings 

From  reviews  of  contractor's  systems,  certain  areas  stand  out  as 
being  troublesome  in  terms  of  meeting  the  C/SCSC.  The  four  most  common 
are:  (a)  inadequate  forward  planning,  (b)  inability  to  plan  and  account 
for  materials  at  the  point  of  usage,  (c)  lack  of  formal  system  procedures, 
and  (d)  undisciplined  budget  practices  (24:93). 

Items  (a)  and  (d)  cause  the  most  trouble  in  measuring  cost  perform¬ 
ance.  If  the  work  statement  has  not  been  properly  expanded  and  planned 
the  question  of  ascertaining  contract  status  becomes  difficult.  Lack  of 
discipline  in  budgeting  is  the  most  common  and  the  most  serious  problem 
area.  Retroactive  adjustments  to  schedules  and  cost  estimates  and 
practices  which  permit  budgets  to  be  shifted  from  one  piece  of  work  to 
another  frequently  result  in  performance  measurement  distortions  which 
render  the  system  useless.  This  practice,  commonly  called  the  "rubber 
baseline"  problem,  tends  to  negate  the  earned  value  concept.  It  follows 
that  establishment  and  maintenance  of  a  valid  performance  measurement 
baseline  which  is  representative  of  contract  accomplishment  is  the  most 
basic  requirement  in  a  performance  measurement  system  (24:93). 

There  are  three  significant  shortcomings  to  the  performance  measure¬ 
ment  subsection  of  SAIMS.  First,  cost  and  schedule  variances  are  measured 
in  value  terms,  namely  dollars.  No  explicit  factors  (milestones)  are 
given  for  schedule  status.  These  variances  are  derived  from  dollar  figures 
as  shown  earlier  and  are  stated  in  dollars.  The  only  calendar  time  reported 
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in  the  CFR  is  the  aggregate  estimate  given  on  line  3  of  the  CPR  Section 
1  summary  (see  Figure  3-5)  and  it  is  not  a  "hard"  number,  i.e.,  it  is 
not  traceable,  but  is  merely  average  dollars  per  day  divided  into  the 
variance. 


Section  1  -  Summary: 

The  A- 10  Program  is  13.5%  complete. 

The  cumulative  schedule  variance  is  (5.3%)  which  represents  an 
improvement  of  2.2%  since  the  last  report.  The  behind  schedule 
position  equates  to  approximately  6  work  days  of  schedule 
variance. 

The  cumulative  cost  variance. is  (4.9%)  which  is  a  degradation  of 
1.6%  since  the  last  report,  is  due  to  direct  and  overhead  rates. 

Both  cost  and  schedule  variances  are  well  within  the  threshold 
tolerance  levels.  The  CPR  Baseline  Schedule  is  based  on  an 
advanced  December  1,  1974  first  flight. 


Figure  3-5.  Extraction  from  the  Oct.  1973  CPR 


Secondly,  all  data  is  presented  in  point  form  implying  a  determin¬ 
istic  connotation.  This  is  obviously  not  true  even  of  performance-to- 
date  information  simply  because  of  the  way  costs  are  allocated  to  work 
in  progress.  When  forecasted  figures  are  given,  e.g>,  Estimate  at 
Completion,  the  uncertainties  involved  in  the  calculations  are  hidden 
by  the  presentation  of  a  single  point  estimate. 

It  is  this  second  shortcoming  which  the  methodology  described  in 
Chapter  4  seeks  to  deal  with.  By  considering  the  time  and  cost  para¬ 
meters  as  random  variables  and  applying  appropriate  probability  theory 
and  simulation  Implementation  it  is  hoped  that  the  future  status  of  the 
contract  will  be  estimated  and  presented  in  a  more  realistic  manner. 
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The  third  shortcoming  was  not  evident  to  the  writers  until  well  into 
the  study.  By  referencing  Figure  3-6  the  reader  will  note  that  the  CPR 
gives  only  top  level  functional  information.  Couple  this  format  with 
the  fact  that  the  CPR  information  is  from  four  to  six  weeks  old  by  the 
time  it  reaches  the  SPO  and  it  becomes  evident  that  the  program  manager 
cannot  rely  on  CPR  information  for  anything  more  than  historical  purposes. 
What  actually  occurs,  at  least  in  the  A-10  program,  is  the  SPD,  like  the 
contractor,  deals  with  problems  functionally.  The  process  of  problem 
propagation,  sometimes  called  the  "rolling  wave"  concept,  occurs  function¬ 
ally,  not  along  WBS  branches.  In  the  A-10  program,  for  example,  the 
first  big  concern  arose  over  late  engineering  drawings.  Because  the 
drawings  were  late,  the  processes  of  tool  planning,  design,  and  fabrica¬ 
tion  were  delayed.  Manpower  accession  problems  in  the  tooling  area 
further  aggrevated  the  problem,  and  so  on.  Succinctly  put,  the  CPR 
should  give  a  belated  Indication  that  something  is  wrong,  but  it  does 
not  tell  where  (see  Chapter  5). 

Summary.  The  SA1M  System  imposes  criteria  upon  defense  contractors 
that  requires  their  management  systems  to  be  capable  of  providing 
various  contract  related  data  elements,  primarily  based  on  cost.  The 
system  suffers  from  two  main  faults)  point  estimation  of  future  variances 
and  presentation  of  data  in  value  terms  (usually  dollars).  Program 
managers  are  consequently  forced  to  seek  alternate  methods  for  predicting 
future  status  and  for  tracking  the  time  variable  explicitly,  (for 
example,  expected  time  in  days  to  the  next  significant  program  milestone 
versus  the  planned  time  to  that  milestone).  The  methodology  explained 
in  Chapter  A  is  one  such  alternative. 
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IV.  Proposed  Methodology 

This  chapter  annotates  the  specific  elements  pertinent  to  the 
risk  assessment  methodology  suggested  by  the  AFA  Risk  Analysis  Study 
Team  (see  quote  on  page  2  )  and  the  assumptions  made  in  applying  the 
methodology  to  the  A-10  Full  Scale  Development  Program.  Basically,  the 
methodology  employs  the  principles  of  network  analysis  and  subjective 
probability  to  obtain  risk  profiles  for  each  of  the  activities  of  the 
project  and  aggregates  them  into  a  risk  profile  for  the  project  as  a 
whole  through  the  use  of  advanced  networking  techniques.  Such  techniques, 
like  VERT  (Venture  Evaluation  and  Review  Technique) ,  use  a  Monte  Carlo 
process  to  repeatedly  sample  the  risk  profiles  of  the  individual 
activities  and  arrive  at  a  distribution  for  the  entire  project.  For 
a  more  detailed  explanation  of  VERT,  see  ADpendix  C. 

To  illustrate  how  this  technique  works  and  show  the  information 
required,  an  example  is  in  order.  Suppose,  for  simplicity's  sake,  we 
have  a  program  made  up  of  two  subsystems,  A  and  B.  In  order  to  complete 
the  project,  A  and  B  must  be  completed  and  then  integrated  and  as¬ 
sembled  ,  activity  C.  This  is  shown  in  Figure  4-1.  Associated  with  each 
of  the  three  activities  are  probability  distributions  for  cost  and 
time.  (The  triangular  distribution  was  assumed  in  the  example.)  In 
reality,  the  project  activities  would  occur  only  once;  however,  using 
network  simulation,  one  can  repeatedly  sample  from  these  distributions 
to  determine  cost  and  schedule  histograms  for  the  total  project.  The 
VERT  output  is  in  the  form  of  probability  density  functions  (pdf)  and 
cumulative  density  functions  (cdf).  The  risk  profile  can  be  obtained 
from  the  output  by  the  formula,  1-cdf.  The  resulting  risk  profile,  say 
for  time,  might  look  something  like  that  shown  in  Figure  4-2.  The 
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area  of  uncertainty  is  between  a  and  b.  As  illustrated,  the  prob¬ 
ability  of  not  completing  the  project  in  the  36th  month  is  0.5. 

The  basic  objectives  in  applying  this  methodology  were  then  to 
(a)  formulate  an  activity  network  of  the  A-10  Full  Scale  Development 
Project,  (b)  obtain  the  best  possible  time  and  cost  estimates  for  the 
network  activities,  and  (c)  use  VERT  to  analyze  the  network  using  pdf's 
derived  from  the  subjective  estimates.  We  hoped  that  this  methodology 
would  provide  the  program  manager  with  information  that  would 
compare  CPR  data  and  afford  lower  WBS  level  traceability  for 
problem  analysis.  Also,  sensitivity  analysis  could  be  performed  by 
varying  the  parameters  of  the  distribution  to  determine  the  effects 
of  different  Inputs  and  to  discover  which  activities  have  the 
greatest  propensity  to  increase  risk. 

The  accuracy  of  this  risk  assessment  methodology  is  primarily 
dependent  on  the  realism  of  the  network  and  quality  of  the  time  and 
cost  estimates.  This  would  suggest  a  very  elaborate  and  detailed  net¬ 
work  and  the  use  of  the  most  advanced  techniques  for  soliciting  sub¬ 
jective  probabilities.  However,  other  requirements  such  as  simplicity 
and  practicality  were  also  germane  to  the  methodology,  not  to  mention 
the  three  month  time  constraint  for  the  thesis  Itself.  A  detailed  net¬ 
work  implies  large  number?  of  activities  which  increases  the  network 
construction  and  maintenance  time  and  also  the  data  collection  efforts, 
•specially  when  subjective  probability  solicitation  is  used.  There  are 
several  elaborate  methods  available  in  the  literature  to  obtain  sub¬ 
jective  probability  estimates  (6:17).  Such  methods  as  the  "You  Bet 
Yourself",  "Fit  Values  to  the  Shspe",  and  "Delphi"  require  considerable 
time  and  administrative  effort  to  obtain  the  distributions.  Therefore, 
it  was  necessary  to  make  some  trade-offs  between  the  requirements  for 
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accuracy  and  those  of  the  real  world. 

Network 

As  mentioned,  the  most  difficult  requirement  for  the  network  was  to 
limit  the  number  of  activities  in  order  to  bound  the  data  collection 
process,  yet  still  produce  a  realistic  network.  We  hoped  to  build  a  net¬ 
work  based  on  the  Work  Breakdown  Structure  explained  in  Chapter  3.  These 
WBS  elements  (level  four  and  above)  provide  the  basic  units  by  which  the 
contractor  reports  cost  and  schedule  information  to  the  Systems  Program 
Office,  the  Air  Staff  and  other  governmental  agencies.  The  contractor 
primarily  monitors  the  program  functionally,  that  is,  he  breaks  the 
project  down  into  functional  tasks  such  as  design,  tooling,  manufacturing, 
quality  assurance,  and  procurement.  These  tasks  are  further  sub-divided 
into  work  packages;  however,  to  construct  a  network  at  this  level  and 
aggregate  to  the  upper  levels  of  the  WBS  would  have  been  a  monumental 
undertaking.  By' basing  the  network  on  the  WBS  elements,  we  thought  we 
could  limit  the  total  activities  to  a  manageable  number.  SPO  and  contractor 
personnel  felt  that  this  type  of  network  was  feasible  and  would  provide 
enough  realism  to  make  the  methodology  valid.  In  addition  to  the  above, 
there  were  three  other  important  requirements  in  establishing  the  network: 
(a)  activities  must  be  clearly  defined  with  start  and  stop  points,  (b) 
relationships  between  activities  must  be  known,  and  (c)  each  activity  must 
have  at  least  one  cost  expert  and  one  time  expert  in  order  to  obtain 
complete  data. 

Since  the  A-10  Full  Scale  Development  Program  covers  the  time  span 
from  March  1973  to  September  1976,  we  decided  to  track  the  program  to 
July,  1975.  This  is  a  significant  milestone  date  in  the  program  as 
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tho  gun  and  aacond  DT&K  aircraft  aro  to  ba  mac ad  and  demonatratad  by 
thla  time.  Thla  declalon  waa  made  for  two  reasons:  (a)  to  limit  the 
number  of  activltiea  in  the  network  and  (b)  to  improve  tho  quality  of 
the  eatimatea  by  not  requiring  the  exparta  to  predict  avanta  too  far 
into  the  future. 

Data 

To  obtain  aubjective  probability  eatimatea ,  it  in  ner.eeeary  to 
aollcit  data  from  knowledgeable  lndividuala  and  fit  thla  data  to  an 
appropriate  probability  dlatribution.  Four  pertinent  quaationa  need 
to  be  answered. 

First,  who  should  provide  the  subjective  input? ?  SPO  peraonnal 
have  some  knowledge  of  the  individual  WHS  elements,  but  their  know¬ 
ledge  of  the  project  is  primarily  dependent  on  data  provided  them  by 
t\.  contractor.  fui*»  prompted  us  to  go  directly  the  contractor  personnel 
lor  our  data  sources.  As  mentioned  in  Chapter  3,  the  contractor's 
management  system  is  functionally  made  up  of  sets  of  cost  accounts  and 

a 

each  cost  account  has  a  manager  who  is  responsible  for  input  data  (BCWS, 
BCWP,  and  ACWP) .  We  thought  that  these  individuals  could  provide  us  with 
the  best  cost  information;  however,  in  a  few  areas,  such  as  Final  Assembly, 
it  would  be  necessary  to  obtain  time  estimates  from  shop  supervisors 
responsible  for  that  particular  task. 

Secondly,  which  subjective  assessment  technique  should  be  used?  At 
pointed  out  in  the  Tripp-Leedom  Report,  the  major  difference  between 
estimates  is  due  not  to  different  assessment  techniques  but  to  the  test 
subjects  themselves  (50:28).  Such  techniques  as  Delphi  and  Standard 
Gamble  require  considerable  time  to  gather  the  data,  while  others  like 
the  Curve  Fitting  techniques  require  it  least  a  basic  understanding  of 
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probability  and  statistics.  in  keeping  with  the  requirements  for 
simplicity  and  practicality ,  we  decided  to  uae  the  three  point 
method  which  ha*  been  used  often  in  the  pant  aa  an  approximating 
technique  in  PERT  (Program  Evaluation  Review  Technique)  (Htl2)»  In 
thie  technique  me  subject  ia  aakad  to  atate  hie  moat  likely  eetimate 
for  coat  and  time  and  hie  lowest  and  higheat  possible  estimates.  These 
three  estimates  aro  then  used  for  formulate  a  cost  and  time  probability 
density  function  for  the  activity. 

Thirdly,  since  the  underlying  distribution  for  the  estimates  is 
unknown,  which  distribution  should  be  assumed?  The  moat  widely  used 
distribution  has  been  the  Beta  distribution  <6 i 97) .  It  has  been  used 
in  the  PERT  methodology  and  Beveihymer  assumed  it  in  his  study  (10:28). 
The  World  Bank  has  compared  several  distributions  fitted  to  subjective 
probability  estimates  in  their  investment  projects  and  found,  in  retro¬ 
spect,  that  the. triangular  distribution  would  have  obtained  an  estimate 
remarkably  close  to  their  actual  results  (41:13).  They  think  that  the 
Beta  distribution  incorrectly  weights  the  value  assigned  to  the  "most 
likely"  estimate.  In  their  experience,  the  most  likely  estimate  is 
not  a  reliable  datum,  and  in  practice  it  often  lies  between  the  value 
with  the  highest  probability  (mode)  and  the  mean.  The  triangular 
distribution  is  simple  and  easy  to  use.  Since  there  is  debate  as  to 
which  distribution  is  more  valid,  we  assumed  the  triangular  distribution 
for  both  cost  and  time.  A  derivation  of  the  triangular  distribution's 
parameters  and  properties  Is  given  in  Appendix  B. 

The  fourth  and  most  difficult  question  to  unswer  is:  How  does 
one  get  the  contractor  personnel  to  provide  honest,  unbiased  opinions 
of  the  cost  and  time  ranges?  This  probably  is  the  most  critical  area  in 
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terms  of  whether  the  methodology  i«  ueeful  or  not.  8inee  tho  coot 
account  manager  la  tho  "export",  hla  estimate  muot  bo  accepted  ao 
tho  moat  valid  aa  long  aa  ho  foola  that  It  oorroaponda  with  hla  boat 
Judgaant.  All  too  of ton  tho  poroonnol  vho  aro  In  the  boat  poaltlon  to 
provide  aound  oatlmatoa  may  alao  have  a  voatod  Interoat  in  diatortlng 
them  (56.96).  For  example,  tho  peraone  being  aokod  their  probability 
aooooomont  nay  not  have  an  incentive  for  eupplying  tho  eatimate  which 
they  boliovo  to  bo  tho  cloooot  to  tho  actual  coat  or  time  rangoo. 

Raaaoni  for  thooo  vagarioo  aro  difficult  to  aacortain  and  control.  Tho 
auaeeaor  may  bo  trying  to  hide  an  internal  deficiency  within  hia  area 
of  roaponalblllty  or  nay  want  to  "pad"  tho  eaUiuate  to  look  good  at  tho 
completion  of  tho  program.  In  thia  aituation,  whore  tho  government 
(8P0)  la  trying  to  obtain  information  from  the  contractor  there  la  even 
more  potential  bias  alnce  the  contractor  may  be  reluctant  to  provide 

outalde  organisations  with  information  that  may  be  adverse  to  hia 

* 

interests.  Woodgata  ouggeata  that  one  possible  way  around  thia  difficulty 
is  to  instill  a  aenae  of  participation  in  the  project  aa  a  whole  by 
fostering  a  team  spirit  and  placing  emphasis  upon  the  overall  project 
objectives  rather  than  individual  department  accomplishments  ( 56*  96) . 

Such  a  sense  of  participation  cannot  be  enforced.  It  is  a  philosophy 
engendered  by  top  management  and  developed  over  a  relatively  long  period 
of  time. 

Another  possible  incentive  for  giving  good  estimates  is  the  use  of 
scoring  rules  or  penalty  functions  (38(1108).  These  methods  pretend 
to  provide  motivation  for  the  expert  to  be  hondst  and  express  his  true 
feelings.  In  the  esse  of  scoring  rules,  the  subject  is  provided  feed¬ 
back  on  how  well  his  estimates  have  correlated  with  actual  results. 
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Penalty  functions  aro  used  much  in  the  same  manner  except  the  subject 
la  penalised  in  some  way  if  his  predictive  values  are  not  within 
predetermined  bounds  of  tho  actual  value.  The  techniques  are  not 
feasible  in  the  weapons  acquisition  process  as  frequent  iterations 
are  required  to  compare  predicted  values  with  actual  results.  The 
outcome  of  a  research  and  development  project  occurs  only  once  and  may 
not  be  known  for  many  months,  if  ever. 

Another  possible  technique  is  a  group  assessment  method  like 
the  Delphi  technique.  A  number  of  experts  anonymously  provide  pre¬ 
dictive  estimates  along  with  justification  for  their  values.  After 
each  response,  the  experts  view  the  group  responses  and  justifications 
and  make  adjustments  to  their  previous  estimates.  After  several 
iterations  of  the  process,  the  group  usually  arrives  at  some  consensus 
opinion.  The  primary  drawbacks  to  the  technique  are  (a)  the  requirement 
for  several  knowledgeable  experts  and  (b)  the  large  amount  of  time 
necessary  to  process  qhe  Iterations. 

The  last  technique  to  be  discussed,  and  the  one  to  be  ased  in 
this  methodology  to  hopefully  motivate  honest,  unbiased  estimates 
is  one  that  the  writers  have  called  the  semi-inducement  method.  Its 
use  has  limited  application  as  there  is  a  requirement  to  have  a 
second  source  of  "semi-experts"  who  have  some  knowledge  of  activity 
times  and  costs.  Such  a  situation  exists  in  the  relationship  between 
the  SPO  and  the  contractor.  The  SPO  personnel  are  charged  with  mon¬ 
itoring  these  variables  and  have  some  knowledge  of  their  status.  The 
questioner  can  use  this  external  information  to  compile  information 
on  each  activity  so  that  he  has  some  prior  knowledge  of  the  ranges 

i 

of  cost  and  time  values  when  he  solicits  them  from  the  primary 
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sources.  This  information  is  used  to  "remind"  the  subject  of  certain 
factors  that  may  have  an  effect  on  his  estimate.  For  example,  if 
the  expert  in  quality  assurance  gave  what  the  solicitor  knew  to  be 
a  low  cost  estimate  for  a  particular  series  of  tests  that  were  part¬ 
ially  compete,  a  reminder  that  his  account  is  presently  x  dollars 
over  budget  cost  and  y  dollars  behind  on  earned  value  might  prompt 
him  to  re-consider  his  estimate;  but,  in  the  final  analysis,  the 
primary  expert's  estimate  can  not  be  totally  rejected. 

Summary 

The  methodology  consists  of  constructing  a  realistic  summary 
network  of  the  A- 10  Full  Scale  Development  Project  up  to  the  time 
the  second  DT&E  aircraft  is  to  be  mated  with  the  30mm  gun.  The  network 
is  based  on  WBS  elements  to  afford  comparison  with  CPR  data  and  reduce 
the  number  of  activities  to  a  manageable  size.  The  data  is  based  on 
the  principles  of  subjective  probability  and  will  be  obtained  from 
contractor  personnel  using  the  three  point  assessment  method.  These 
estimates  are  then  fitted  to  the  triangular  distribution  to  obtain 
the  required  time,  and  cost  probability  density  functions  for  each 
of  the  activities.  The  risk  profiles  for  the  individual  activities 
are  computed  and  aggregated  using  the  VERT  simulation  routine  to 
obtain  overall  risk  profiles  for  schedule  and  cost. 

Further  References 

It  should  be  noted  that  the  areas  of  network  analysis  and 
subjective  probability  estimates  are  quite  vast  and  controversial. 

This  thesis  in  no  way  has  addressed  all  the  alternatives,  assumptions, 
and  limitations  associated  with  these  two  principles.  For  those 


34 


GSA/SM/74-1 


V.  A  Test  Application 


Constructing  the  Network 

The  first  step  in  applying  the  methodology  was  to  construct 
a  simplified,  but  realistic,  UBS  network  of  the  A- 10  project. 

Although  neither  the  Contractor  nor  the  SPO  was  using  a  PERT  pro¬ 
ject  network,  it  was  hoped  that  the  Contractor's  Management  Network, 
a  portion  of  which  is  depicted  in  Figure  5-1,  could  be  used  to  con¬ 
struct  a  network  suitable  for  this  application.  Further  study  of  the 
Management  Network  revealed  that  its  activities  are  not  clearly 
defined.  For  example,  in  Figure  5-1,  top  line,  there  is  a  node  in¬ 
dicating  completion  of  Design  Layout,  but  we  have  no  idea  from  the 
network  when  the  activity,  "Design  Layout"  was  started.  In  the  second 
line,  it  is  undetermined  as  to  whether  Tool  Pre-planning  is  completed 
at  the  start  of  Tool  Planning  or  whether  these  two  activities  are 
concurrent  for  a  period  of  time.  It  is  primarily  an  event  oriented 
network  with  nodes  representing  contract  milestones  and/or  functional 
events.  Initial  attempts  to  identify  the  activities  implicit  in  the 
event  network  and  determine  interdependencies,  indicated  that  the 
resulting  network  would  include  too  many  detailed  activities,  making 
the  application  Impractical  under  the  constraints  of  the  research. 

A  more  simplified  network  was  constructed  using  the  Contract 
Work  Breakdown  Structure  Dictionary,  a  contractor  document  which  lists 
the  WBS  elements  to  as  low  as  the  sixth  level  and  provides  the  link 
between  these  elements  and  the  functional  structure  via  cost  account 
aggregation.  This  network,  A  portion  of  which  is  shown  in  Figure  5-2  > 

showed  promise  of  meeting  the  requirements  and  constraints  annotated 

i 

in  Chapters  2  and  4.  Both  the  Contractor  and  SPO  advisors  expressed  con- 
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Figure  5-1  Contractor's  Management  Network 
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fidence  in  the  potential  realism  of  the  network.  Before  finalizing 
the  network  and  coding  it  into  the  VERT  simulation  routine,  a  pre¬ 
liminary  data  collection  effort  was  made  to  insure  that  the  data 
sources  could  provide  the  necessary  estimates  for  each  of  the 
activities.  A  portion  of  the  network  was  selected  for  the  trial  run. 
This  first  attempt  in  obtaining  time  and  cost  estimates  surfaced 
major  problems  in  the  application  of  the  methodology. 

The  major  problem  was  correlating  the  data  sources  with  the 
activities.  In  short,  we  were  unable  to  find  experts  within  the 
Contractor's  organization  who  could  provide  aggregate  time  or  cost 
estimates  for  an  entire  WBS  activity.  The  crux  of  the  problem  lies 
in  the  relationship  between  WBS  and  functional  structures.  This 
relationship  is  illustrated  in  Figure  5-3.  As  depicted,  a  number  of 
functional  departments  contribute  to  the  construction  of  the 
Forward  Fuselage  section.  Also,  the  activities  of  these  different 
departments  are  intertwined  with  activities  dependent  upon  other 
activities  before  start  or  completion  can  occur.  For  example,  look¬ 
ing  at  Figure  3-3  again,  the  functional  tasks  of  design,  tooling, 
and  procurement  are  related  in  that  some  of  the  design  effort  must 
be  completed  before  tool  planning  or  procurement  operations  can  be¬ 
gin.  Also,  all  the  engineering  design  must  be  completed  before  the 
tooling  activity  can  >e  ^nishoU  It  is  very  difficult  to  determine 
just  exactly  how  far  along  on*  activity  must  be  before  another  can  be¬ 
gin.  In  order  to  determine  Information  of  this  detail,  one  would  have 
to  network  the  project  at  the  work  package  level.  There  are  approx¬ 
imately  3000  work  packages  in  FRC's  portion  of  the  program.  This 
WBS/functional  relationship  is  prevalent  throughout  most  of  the  other 
WBS  elements.  In  order  to  obtain  cost  and  time  estimates,  the  expert 
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must  have  sufficient  knowledge  of  the  different  functional  tasks 
that  are  necessary  for  the  completion  of  a  particular  WBS  element. 

Since  the  contractor  manages  through  functional  organization,  e.g., 
he  "tracks"  the  project  through  the  planning  and  control  of  his  engin¬ 
eering,  tooling,  quality  assurance,  manufacturing,  material,  and  sub¬ 
contracting  functions,  few  of  the  contractor’s  middle  management 
personnel,  such  as  the  cost  account  managers,  are  in  a  position  to 
know  how  much  other  functions  contribute  to  a  particular  WBS  element. 
They  are  very  knowledgeable  in  their  specific  functional  area  but  do 
not  have  the  breadth  of  knowledge  necessary  to  express  cost  or  time 
estimates  for  an  entire  WBS  activity. 

The  alternatives  to  this  dilemma  were  to  (a)  find  suitable  experts 
knowledgeable  of  the  existing  WBS  activities  or  (b)  sub-divide  the 
activities  into  specific  functional  cost  accounts  and  thus  permit  the 
cost  account  managers  to  make  judgements  on  those  tasks  within  their 
area  of  responsibility.  With  the  exception  of  a  few  elements  like  Final 
Integration  and  Assembly,  Quality  Assurance,  and  Procurement,  contractor 
personnel  knowledgeable  of  the  WBS  activities  were  non-existent.  One 
can  easily  see,  changing  from  a  WBS-'oased  network  to  a  functional  net¬ 
work  causes  the  network  to  explode.  As  depicted  In  Figure  5-3,  the 
number  of  activities  Increases  from  1  to  32  (19  activities  and  13 
transportation  arcs).  The  initial  WBS-based  network  contained 
approximately  70  activities,  most  of  which  increase  by  a  conservative 
factor  of  20  if  a  functional  network  were  constructed.  The  resultant 
network  would  have  contained  about  1400  activities.  The  data 
collection  process  would  then  have  been  enormous. 
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Soliciting  Estimates 

Further  problems  were  encountered  when  we  attempted  to  solicit  time 
estimates  for  the  final  assembly  phase.  Although  the  assembly  task  had 
been  planned  in  detail,  the  production  planner  in  charge  of  the  opera¬ 
tion  indicated  that  deviations  from  the  plan  would  occur,  especially 
during  the  first  few  DT&E  aircraft.  Some  parts  would  not  be  available 
on  time  and  problems  would  occur  requiring  re-design  or  re-fabrication. 
However,  he  maintained  that  the  assembly  operation  would  not  come  to 
a  standstill  just  because  the  next  step  in  the  plan  could  not  be  completed. 
"Work-around  plans"  would  be  implemented,  circumventing  problem  areas 
and  rescheduling  them  out  of  sequence.  He  did  not  know  exactly  how  many 
work-around  operations  would  be  required  but  did  mention  that  over  a 
hundred  was  not  unlikely.  Due  to  these  potential  work-around  require¬ 
ments,  he  was  very  reluctant  to  commit  himself  to  estimates  for  the 
assembly  activities.  For  example,  according  to  the  plan,  the  three 
fuselage  sections  are  to  be  fully  "stuffed"  with  appropriate  hydraulic 
and  electrical  components  before  the  sections  are  mated.  The  production 
planner  was  reluctant  to  commit  himself  to  this  sequence  of  activities. 

He  argued  that  the  mating  operation  could  take  place  earlier  if  some  of 
the  hydraulic  or  electrical  components  were  not  available  on  time.  These 
late  components  would  then  be  assembled  at  a  later  point  in  the  assembly 
operation.  Some  contingency  plans  were  already  being  formulated  at  the 
time  of  this  interview  since  it  was  already  known  that  a  few  long-lead 
items  would  most  likely  be  late. 

The  existence  of  these  possible  work-around  operations  causes 
further  woes  for  the  networker.  Because  numerous  changes  arc<  expected 
in  the  network  activities  during  the  construction  of  the  first  few 
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DT&E  aircraft,  frequent,  perhaps  daily,  updating  will  be  necessary 
in  order  to  maintain  currency.  Since  the  network  was  to  be  main¬ 
tained  by  the  SPO,  it  would  have  required  a  great  deal  of  coordination 
between  the  contractor  and  SPO  to  maintain  a  realistic  network. 

Bias  Problems 

As  expected,  but  to  a  greater  degree  than  anticipated,  the  problem 
of  soliciting  unbiased  estimates  was  encountered.  As  mentioned  in 
Chapter  4,  we  hoped  to  reduce  the  bias  by  gathering  as  much  cost  and 
time  information  available  about  each  activity  prior  to  interviewing 
the  experts  to  obtain  their  predictive  estimates.  In  two  areas  of 
quality  assurance  we  were  able  to  find  contractor  personnel  capable  of 
providing  cost  estimates  for  two  WBS  activities.  In  both  of  these  cases, 
their  initial  estima  es  were  considerably  lower  than  we  had  anticipated. 
The  Cost  Performance  Report  indicated  that  these  two  accounts  werj 
approximately  15%  over  their  cumulative  budget- to-date,  but  would  be  on- 
target  at  the  end  of  the  program.  Information  from  SPO  personnel 
indicated  that  the  overrun  was  greater  than  the  CPR  information  and  that 
it  was  highly  unlikely  that  they  could  be  reduced  significantly.  After 
over  a  half  hour  of  discussion,  both  cost  account  managers  revised  their 
estimates  to  higher,  more  realistic  values.  They  later  admitted  that 
they  were  very  reluctant  to  provide  individuals  outside  the  organization 
with  information  that  deviated  from  that  reported  through  the  formal 
reporting  system,  i.e.,  the  CPR.  The  fact  that  we  appeared  to  have 
accurate  information  about  their  accounts  in  the  pertinent  WBS  activities 
prompted  them  to  revise  their  estimates. 

It  appears  that  the  semi-inducement  technique  has  some  potential  to 
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reduce  grot*  bias  in  the  estimates  but  it  also  has  major  shortcoming*. 

It  tan  only  be  applied  to  those  situations  where  external  information 
in  available.  Sven  if  the  prior  information  is  available,  substantial 
research  effort  is  required  to  assimilate  the  information.  Using  a 
network  approach,  one  must  limit  the  number  of  activities  to  avoid  an 
expensive  and  time  consuming  data  collection  process.  There  is  elan 
the  possibility  that  the  technique  itself  might  bias  the  estimates. 

An  aggressive  interviewer  could  possibly  Influence  the  expert  to 
estimate  values  inconsistent  with  his  own  beliefs.  The  number  of 
samples  taken  were  too  small  to  prove  or  disapprove  this  possibility. 

Summary  * 

We  were  unable  to  construct  a  WBS-based  network  that  was  consistent 
with  the  network  requirements  established  for  the  mothodology.  Possible 
work-around  operations  pertinent  to  the  construction  of  the  first  few 
development  aircraft  further  complicated  network  construction,  mainten¬ 
ance,  and  data  collection.  Finally,  the  efforts  made  to  solicit  cost 
and  time  estimates  Indicated  considerable  reluctance  on  the  contractor's 
part  to  provide  us  with  unbiased  estimates.  The  magnitude  of  these 
problems  prompted  us  to  abandon  the  methodology  as  impractical  and  seek 
alternate  ways  to  assist  the  SPO  in  assessing  contractor  performanca. 
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VI.  least  Squares  Forsessting 

Subaaqusnt  to  concluding  that  our  application  of  tho  original 
methodology  on  a  seal «  detailed  enough  to  afford  realism  wee  not  feasible 
in  the  time  dieted,  we  sought  other  ways  to  foreoast  coat  and  schedule 
performance.  Time  prevented  no  from  investigating  all  considerations  so 
we  pursued  two  approaches  bated  on  laaat  squares  regression  eialysis. 

This  chapter  reflects  those  efforts. 

MaUY*tl9,n 

Motivation  arose  from  obaerving  the  monthly  CFR  data,  specifically 
the  "At  Completion"  column.  By  referencing  Figure  6-1,  tho  reader  will 
note  that  the  Estimate  at  Completion  is  precisely  equal  to  Budget  et 
Completion  although  there  exist  several  cases  of  adverse  cost  variance 
(Indicated  by  parenthesis)  in  the  Cumulative  to  Date  Section.  This  imp¬ 
lies  that  for  those  elements  with  adverse  coat  variance  (overrun)  the 
contractor's  estimate  to  completion  wee  arrived  at  by  "backing  in."  l.e.. 
somehow  he  will  perform  the  remaining  work  for  exactly  the  current  adverse 
variance  dollars  less  than  Is  budgeted. 

Both  tho  contractor  and  the  SPO  acknowledged  this  was  not  posalble 
in  soma  oases  and  the  contractor  was  admonished  to  oorreot  the  situation. 
The  December  1973  CPR  is  the  first  to  show  non-sero  varisnees  at  completion 
but  there  a till  is  a  question.  Those  non-sero  varisnees  at  completion 
era  just  squsl  to  currant  (December)  cost  varianoea.  as  in  Figure  6-2. 

The  contractor  simply  sddad  currant  variance  to  Budget  at  Completion  to 
arrive  at  Estimate  at  Completion.  Given  thia  sequence  of  events,  ve 
questioned  whether  or  not  the  contractor  was  really  attempting  an  accurate 
forecast.  The  FRC  Management  Control  and  Information  System  (MC1S) 

Training  Manual  states i 
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The  [  Estimate  to  Completion  ]  ETC  is  a  fresh  objective  re-analysis  of 
the  remaining  work  to  be  done  independent  of  the  budget  or  actual 
costs  to  date...  Particularly  if  the  ETC's  are  larger  than  expected, 
analysis  and  ETC's  at  the  Work  Package  level  should  be  performed  to 
pinpoint  the  source  of  the  problem  (17:135). 

At  the  suggestion  of  members  of  the  Program  Control  Directorate  in 
the  A- 10  SPO,  we  decided  to  try  some  simple  regression  analysis  as  a 
first  approximation  to  improving  the  CPR  variance  forecasting.  It  is  well 
to  note  again  that  SAIMS  schedule  data  is  measured  in  dollars,  not  days. 
The  Linear  Model 

We  first  chose  to  attempt  an  analysis  on  the  cost  account  level  with 
the  option  of  summing  those  numbers  and  repeating  the  process  on  some 
higher  level,  either  WBS  or  functional.  Toward  that  end  we  obtained  data 
from  two  FRC  MCIS  reports,  the  Project  Cost/Schedule  Comparison  Report  and 
the  Planning  Package  Status  Report,  for  months  May  through  November  1973. 
Although  the  contract  was  signed  on  1  March  1973,  the  first  SAIMS  data 
did  not  result  until  the  month  of  May,  hence  the  two  month  lag.  The  model 
used  was  the  same  for  both  time  and  cost  variances,  as  well  as  for  direct 
labor  hour  variances  (an  additional  data  item  obtainable  from  the  two 
reports) .  Since  there  can  be  no  variance  prior  to  contract  initiation, 
we  omitted  the  Intercept  term  and  used  an  equation  of  the  form 

y±  «  BXj^  +  ei 

where  is  the  observed  variance  on  data  set  i,  x^  is  the  independent 

variable  (months  or  BCWS)  for  set  i,  B  is  the  parameter  (coefficient) 

to  be  estimated,  and  e ^  is  the  measurement  error  for  set  i.  The  e's 

2 

are  assumed  to  be  independent  with  mean  zero  and  variance  sigma  ,  a 
constant.  For  analysis  of  variance  purposes,  the  e's  are  also  assumed 
to  be  normally  distributed  (here  we  mean  variance  in  the  statistical 
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sense) . 

Since  some  cost  accounts  did  not  open  at  contract  initiation 
we  had  at  most  seven  data  points.  We  concluded  that  the  Full  Scale 
Development  effort  would  have  been  too  large  (313  cost  accounts)  so  we 
limited  the  analysis  to  the  Basic  Structure  element  defied  in  Chapter 
2.  Further,  since  most  recurring  cost  accounts  (those  that  would  have 
entries  periodically  for  each  of  the  10  DT&E  aircraft)  were  not  open, 
we  omitted  them.  Through  an  oversight  we  also  ommitted  two  non-recurring 
cost  accounts.  The  result  was  that  we  had  37  cost  account  data  sets. 

Each  set  contained  BCWS,  BCWP,  ACWP,  Budgeted  Labor  Hours,  and  Actual 
Labor  Hours.  We  also  obtained  BCWS  numbers  for  the  relevant  cost  accounts 
through  June  1974.  A  typical  data  set  is  shown  in  Table  6-1. 


BCWS 

BCWP 

ACWP 

BUDGETED  LABOR  ACTUAL  LABOR 

Month 

HOURS 

HOURS 

62600 

62599 

61490 

4198 

3757 

May  73 

103819 

99047 

93633 

6646 

5886 

Jun 

192794 

173018 

186036 

11622 

12250 

Jul 

367333 

294932 

374721 

19817 

25059 

Aug 

562132 

491659 

616583 

33048 

41532 

Sep 

580413 

564559 

760687 

37977 

51144 

Oct 

645574 

645812 

851250 

43481 

57118 

Nov 

686685 

Dec 

712015 

Jan  74 

741217 

Feb 

757903 

Mar 

1012251 

Apr 

1012251 

May 

1012251 

Jun 

Cost  Account  -  019B 

-  Engineering, 

Fuselage  I&A 

Table  6-1.  Typical  Data  Set 

Implementation.  We  used  the  OMNITAB  program  on  the  GE  600  series 
computer  to  perform  the  analysis  (29).  By  arranging  the  data  sets  in 
the  OMNITAB  worksheet  we  were  able  to  write  a  simple  program  (Appendix 
D)  to  do  the  arithmetic  and  perform  the  regressions.  We  also  used  the 
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automatic  plot  feature  of  OMNITAB  to  print  the  graphs  in  Figures  6-3 
through  6-  6.  Figure  6-6  is  a  prediction  similar  to  Figure  3-4. 

Findings .  The  results  of  the  program  were  questionable.  First, 
the  omission  of  the  two  non-recurring  accounts  and  all  the  recurring 
accounts  made  the  totals  lower  than  those  reported  on  the  CPR.  For 
instance,  the  November  CPR  listed  the  Basic  Structure  cumulative-to-date 
BCWS  as  $7,147,000  (to  the  nearest  thousand  dollars)  and  our  data  showed 
$5,727,614.  By  adding  those  cost  accounts  omitted  from  our  data  we  could 
show  a  November  BCWS  of  $6,997,636.  We  cannot  explain  the  149  thousand 
dollar  difference. 

Secondly,  our  cost  variance  prediction  routine  was  necessarily  a 
two  step  procedure  and  hencethe  linearity  assumption  was  suspect.  The 
reader  will  recall  from  Chapter  3  that  cost  variance  equals  earned  value 
minus  actual  costs.  This  implies  that  a  future  estimate  of  actual  cost 
must  be  the  sum  of  a  predicted  earned  value  and  a  predicted  cost  variance. 
To  find  the  predicted  earned  value,  we  regressed  earned  value  against 
BCWS  as  the  independent  variable.  To  find  predicted  cost  variance,  we 
regressed  cost  variance-to-date  against  months  one  through  seven.  Table 
6-2  lists  the  equations.  Table  6-3  lists  a  summary  of  the  results.  The 
large  95%  confidence  intervals  emphasize  the  uncertainty  of  the  process. 

Thirdly,  we  were  skeptical  of  the  information  displayed  in  Figures 
6-3.  and  6-4.  Note  the  simultaneous  decrease  of  cost  and  schedule  variance. 
Generally  when  a  contractor  is  behind  schedule,  he  must  spend  dollars 
(overtime)  to  catch  up,  but  this  did  not  happen  is  this  case.  The  only 
possible  explanation  within  the  C/SCS  Criteria  is  that  a  peculiarly  large 

block  of  earned  value  was  accrued  in  November.  Regardless,  the  data  jump 

i 

I 

reduced  the  reliability  of  a  linear  model. 
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Figure  6*4.  Actual  and  Predicted  Cost  Variance 
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OMNITAB  PROSHAM  TOR  A-ll  COST  ACCOUNT  VARIANCE  REGRESSIONS 


A B S*  MONTHS 

ORD-'BCHS  MAT-JUN  (.),  EARNEO  VALUE  <•>,  BCMP*CST  VAR  <*>, 
NO.  or  'MS.  PLOTTEO  42  NO.  NOT  PLOTTED  <OUT  OF  BOUNDS) 


B.IOItE  07*  OCHS  MAT* JUN  EARNED  VALUE  BCHP*CST  VAR 


l.OOOOE  IT* 


I. TIME  IT* 


B.llllt  1C* 


4.IIIIE  It* 


Sltlt6.ll 
124640,10 
1093943.0 
1555636.0 
24TB821 . 0 

3776038.0 

572/614.0 

•707624.0 

11641646. 

12202636. 

14324162. 

16137931. 

17901110. 

18619421. 


306337. •• 
541625. 00 
634045.00 
1262911.0 
2236441.0 

33535  JO ,  0 
54168U7.0 
7901544.9 
9764319.4 
11166165. 

13)29)03. 

1470I7HB, 

16418627. 

17167060. 


346958. II 
581192.08 
893693.01 
1459269.0 
2715465,8 

4141486.1 
6131134.0 
8736097 • 9 
10603191. 
12126356. 

14167413. 

15832618. 

17634776. 

18387517. 


/'  / 
// 

if 


J/t 

'At 


*  ,  *64*0  •• 


May  73 


Nov  73 


I. 8**78  81 


1,418*1  It 

Jun  74 


Figure  6-6.  Predicted  Performance  Through  June  1974 
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Last,  there  is  a  peculiarity  in  the  data  due  to  the  difference  be¬ 
tween  the  planned  work  package  budget  and  the  released  work  package  bud¬ 
get  (released  budgets  indi  ate  that  the  work  package  has  been  opened  (or 
it  is  less  than  two  weeks  from  being  opened) .  Although  this  is  not  always 
the  case,  when  it  occurs  it  can  have  a  distorting  effect  on  the  cost 
account  numbers,  particularly  when  it  is  realized  that  we  used  a  combina- • 
tion  of  the  two  types  of  work  package  budgets  to  get  the  BCWS  numbers 
to  June  1974.  Essentially  whot  happens  is  that  unopened  work  packages 
may  be  replanned  and  rebudgeted  (within  the  confines  of  the  appropriate 
cost  account  budgets)  up  to  two  weeks  before  the  scheduled  opening  date. 
This  has  the  effect  of  altering  the  shape  of  the  cost  account  cumulative 
BCWS  curve  and  hence  alters  the  independent  variable  in  the  earned  value 
regression. 

Summary .  Despite  these  drawbacks,  this  approach  or  one  similar  to 
it  should  be  better  than  what  is  now  being  done  in  the  A-10  program  CPR, 
especially  for  near  term  predictions  (like  the  7th  month  point  we  used). 

It  also  offers  the  advantage  of  being  able  to  look  at  the  problem 
functionally  if  desired,  because  the  data  is  at  the  cost  account  level. 

The  Gompertz  Model 

The  advisors  in  the  A-10  SPO  suggested  that  the  cumulative  BCWS 
curve  might  have  a  distribution  similar  to  that  of  the  Gompertz  Curve 
which  has  often  been  used  in  the  area  of  technical  forecasting  (34:113). 
The  equation  for  the  Gompertz  Curve  is: 


Y  ■  Le 


-be 


-kt 


It  ranges  from  zero  at  t  equals  minus  infinity  to  the  upper  limit,  L, 
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at  t  equals  plus  infinity,  where  t  is  the  independent  variable.  However, 
the  curve  is  not  symmetrical.  The  inflection  point  occurs  at  t  =  (In  b) /k, 
where  Y  -  L/e.  Figure  6-7  shows  a  Gompertz  Curve  for  which  L,  b,  and  k 
are  all  equal  to  one. 


Implementation.  We  used  the  GROCRV  program  on  the  CDC  computer  to 
perform  the  Gompertz  regressions  (see  Appendix  E).  We  first  regressed  the 
Gompertz  Curve  against  the  projected  cumulative  BCWS  for  the  total  A-10 
Development  Program.  As  shown  in  Figure  6—8,  the  Gompertz  Curve  appears 
to  "fit"  the  cumulative  BCWS  curve  fairly  well.  The  Coefficient  of 
Determination  using  a  L  value  of  125  million  dollars  is  0.99834.  If  one 
further  assumes  that  the  ACWP  cumulative  curve  has  a  similar  shape,  then 
the  Gompertz  Curve  could  possibly  be  used  to  predict  future  ACWP  values 
using  data  from  the  CPR. 
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Am  with  tha  linear  regression  medal ,  ui\ly  tha  cumulative  aeoeuuu 
from  Che  H**l«  Btrueture  WltH  element*  were  used,  Since  the  data  fret* 
the  two  KRIS  report*  wa*  auapeet*  we  used  CPK  date  for  this  effort, 
live  data  eat  .tneluded  eight  meet  ha  of  eumulativ*  ACWP  number*  and  a 
rente  of  poaaible  ACWP  value*  for  the  Ultimate  at  Completion!  We  uaed 
the  3Bth  ualendar  month  a*  the  terminating  month  ainee  Baeio  Structures 
are  to  be  completed  by  thir  time.  By  varying  the  ACWF  values  at 
completion*  we  constructed  a  family  of  predicted  curve*  that  might  possibly 
represent  the  actual  dollar  expenditure*  (in  million*  of  dollar*)  tor 
the  Basic  Structure*  element  during  the  program  (see  figure  6-9).  Fitting 
the  Gompert#  curve  to  theee  nine  point*  (the  eight  data  point*  and  the 
eatimuted  end-of-program  ACWP)  resulted  in  vary  high  Coefficient*  of 
Determination  for  ell  the  regressions.  They  ranged  from  0.99932  to 
0.99963.  The  upper  limit*  L,  waa  adjusted  through  trial  and  error  to 
obtain  the  highest  Coefficient  of  Determination  for  each  data  aet. 

find inn*.  Aa  one  can  aee  by  looking  at  Figure  6-9,  all  the  curvea 
ere  el  moat  superimposed  on  one  another  during  tho  early  stages  of  the 
program.  They  do  not  begin  to  diverge  until  about  the  10th  month.  At 
the  12th  month  the  divergence  ip  ample  to  begin  comparing  CPR  data  for 
that  month  with  the  predicted  value*  and  determine*  possibly,  which  of 
the  actual  coat  curvea  the  contractor  appears  to  be  on,  Aa  the  program 
progresses  and  additional  data  becomes  available*  the  curves  could  be 
up-dated  and  some  of  the  outlying  curve*  could  be  eliminated  aa  infeasible. 

The  inflection  points  of  the  Gompert*  curves  fitted  to  the  BCWS 
cumulative  data  and  ACWP  data  could  be  compared  to  provide  another  means 
of  estimating  which  actual  curve  the  contractor  was  following.  This  could 
be  used  as  early  as  the  14th  month  since  the  inflection  point  appears  to 
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Figure  6-9  Predicted  (Gompertz)  Cost  Curves  for  Basic  Structures 
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occur  About  that  time  in  the  program. 

Summary.  the  assumptions  necessary  for  this  methodology  ere  shaky 
at  beat*  Sven  if  the  Comparts  Curve  does  fit  the  UCWS  curve  reasonably 
well  *  there  is  no  guarantee  that  the  AGWP  curve  will  be  similar  to  the 
BCW8  curve.  Problems  encountered  in  the  program  could  cause  a  rapid 
shift  from  one  predicted  curve  to  another  and  even  result  in  multiple 
inflection  points  in  the  actual  curve.  Such  a  case  can  easily  occur 
in  tha  A-10  program  because  the  contract  ia  being  renegotiated  aa  a 
raault  of  funding  alterations  by  Congress. 
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VIZ.  Conclusions  and  Rocommondetiona 

Xn  tha  baginning  of  the  atudy,  we  thought  wa  had  a  well  bounded 
problem  auitable  for  a  scientific  management  approach;  however,  aa  we 
laarnad  more  about  tha  waapona  acquisition  proceso  and  management  within 
tha  System  Program  Office,  wa  discovered  that  tha  weapons  acquisition 
process  is  complex,  dynamic,  and  fraught  with  many  sources  of  uncertainty. 
Rven  though  wo  were  unable  to  implement  the  proposed  methodology,  we  feel 
that  the  learning  process  experienced  allows  ua  to  draw  some  conclusions 
concerning  the  risk  assessment  approach  taken  here  and  to  make  a  few 
inferences  about  risk  analysis  in  general  in  the  process  of  buying  weapons. 
Finally,  we  have  proposed  some  recommendations  which  we  hope  will  be  of 
some  benefit  to  others  who  might  contemplate  study  in  this  area.  We 
caution  that  the  contents  of  this  chapter  are  based  mostly  on  our  opinions 
which  in  turn  arc  the  result  of  a  sample  size  of  one,  namely  the  A-10 
Full  Scale  Development  Program. 

Particular  Conclusions  and  Recommendations 

1.  Conclusions:  Although  not  a  risk  assessment  in  the  strict  sense 
of  the  AF  Academy  Study  Team’s  definition,  the  methodology  proposed  in 
this  study  could  provide  the  contractor  and  government  with  useful  infor¬ 
mation. 

The  AF  Academy  Study  imp. les  that  risk  assessment  is  a  component  of 
risk  analysis,  that  is,  the  first  step  in  risk  analysis  is  to  determine 
the  risk  associated  with  each  of  the  possible  alternatives.  In  this 
case  study  there  was  only  one  basic  alternative,  that  of  developing  the 
A-10  aircraft.  Also,  to  be  considered  a  true  risk  assessment,  all  three 
variables,  i.e.,  cost,  time,  and  technical  performance,  should  be  assessed 
to  obtain  the  "program"  risk.  For  reasons  explained  in  Chapter  2,  we 
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excluded  the  performance  variable.  However,  we  feel  that  the  methodology 
could  still  provide  useful  cost  and  schedule  Information.  Its  main 
attribute  Is  the  presentation  of  cost  and  time  estimates  In  a  probabilistic 
format,  rather  than  as  point  estimates.  The  manager  Is  provided  a  range 
of  possibilities  from  which  implications  of  vnc-.  ruainty  can  be  drawn  for 
each  activity,  and  the  total  program,  by  oLs~-*\xng  the  statistical  variance 
In  the  resultant  distributions. 

Since  there  is  only  one  alternative,  the  methodology  Is  essentially 
an  Information  system.  By  using  the  principles  of  management  by  exception, 
the  AF  program  manager  can  identify  areas  with  high  "estimate"  uncertainty 
and  high  probability  of  not  obtaining  planned  cost  and  schedule  objectives, 
and  thus  devote  most  of  his  attention  to  those  areas.  However,  the 
information  will  not  tell  him  what  is  wrong.  There  are  three  possibilities: 
(a)  the  initial  plan  was  in  error  (possibly  due  to  target  or  process 
uncertainty) ,  (b)  the  plan  is  being  poorly  executed  (internal  program 
uncertainty  can  cause  this)  or,  (c)  the  data  is  inaccurate,  (most  likely 
due  to  bias  in  the  estimates  or,  in  this  case,  possibly  to  the  exclusion 
of  the  performance  variable).  Logically,  the  manager  would  seek  to 
determine  the  cause  or  causes.  If  he  then  acted  to  correct  those  causes, 
the  methodology  could  be  considered  a  type  of  control  system.  The 
implications  of  this  deduction  will  be  discussed  later. 

Recommendation.  If  the  probabilistic  format  is  considered  useful, 
then  perhaps  the  Cost  Performance  Report  could  be  altered  to  present  the 
data  in  this  way  at  substantially  less  cost  than  would  be  incurred  using 
this  network  methodology. 

2.  Conclusion:  A  technique,  like  that  attempted  in  this  study,  which 
employs  network  analysis  and  subjective  probability  techniques  should 
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not  be  attempted  without  a  strong  commitment  by  both  the  SPO  and  the 
contractor. 

The  methodology  is  very  similar  to  that  of  PERT,  except  for  the 
stochastic  treatment  of  data.  Like  PERT*  the  program  should  be  applied 
early  in  a  development  effort  with  the  full  support  of  those  who  are 
going  to  use  and  maintain  the  system.  Otherwise,  there  may  only  be 
a  superficial  desire  to  see  the  system  work,  and  it  may  become  just 
an  ancillary,  bureaucratic  function.  As  suggested  by  the  AF  Academy 
Study,  it  should  be  implemented  by  an  interdisciplinary  team  made  up 
of  trained  analysts  in  mathematics,  probability,  statistics,  operations 
research,  and  computer  science  -  aided  by  cost  analysts,  production, 
design,  and  engineering  people,  and  experts  in  various  technical  disciplines 
(33:6).  The  cost  of  this  approach  would  be  high,  but  we  see  no  other  way 
to  reliabily  test  its  feasibility.  Unfortunately  we  did  not  have  the 
time  nor  the  resources  necessary  to  take  this  normative  approach.  Further, 
neither  the  contractor  nor  the  SPO  was  willing  to  incur  the  cost  of  such 
an  elaborate  management  tool  considering  the  relative  simplicity  of  the 
aircraft  to  be  built. 

Recommendation :  If  both  industry  and  government  mutually  determine 
that  this  type  of  methodology  has  potential  merit  for  an  individual 
program,  then  they  should  commit  the  necessary  resources  to  insure  a 
reasonable  degree  of  success.  In  our  opinion,  it  is  doubtful  that  this 
type  of  methodology  can  be  successfully  implemented  unilaterally  by  the 
Air  Force  on  a  low  key,  low  cost  basis. 

3.  Conclusions:  Successful  use  of  this  type  of  methodology  as  a 
control  system  is  unlikely.  As  indicated  in  the  first  conclusion,  there 
may  be  a  tendency  for  the  Air  Force  to  use  the  system  as  a  means  to 
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monitor  and  control  contractor  performance.  Since  the  input  data  for 
the  methodology  is  obtained  subjectively  from  personnel  in  the  contract¬ 
or's  organization)  its  use  as  a  control  system  by  the  AF  will  most  likely 
increase,  the  estimate  bias  as  a  result  of  adverse  feedback  reactions. 

These  reactions,  both  intentional  and  unintentional,  may  suppress  the 
flow  of  accurate  information  between  the  contractor  and  the  SPO.  Within 
his  own  organization  even  the  contractor  may  have  sotve  difficulty  in 
getting  the  experts  to  accurately  transmit  their  perv  lotions  upward  to 
their  superiors,  but  at  least  the  contractor  has  some  control  over  this 
internal  bias  via  the  infusion  of  a  compensating  management,  philosophy. 

If  this  methodology  is  used  by  the  SPO  to  correct  contractor  performance, 
the  contractor  may  be  reluctant  to  provide  unfavorable  information, 
especially  in  a  situation  like  the  A-10  where  a  company's  survival  as  a 
prime  contractor  may  depend  on  an  impending  production  decision. 

Recommends t ion :  If  the  information  from  this  type  of  methodology 
is  to  be  supplied  to  the  AF  program  manager,  extraordinary  measures  should 
be  taken  to  avoid  using  the  system  as  a  control  device,  lest  it  be 
rendered  useless  by  adverse  feedback  reaction. 

4.  Conclusion:  SAIMS  in  the  A-10  program  has  two  major  short¬ 
comings:  (a)  as  presently  determined,  the  time  variance  (given  in  dollars) 
is  inaccurate  and  misleading,  and  (b)  although  required  in  the  contract 
via  the  CSCS  Criteria,  the  contractor  is  not  supplying  the  SPO  with 
sincere  cost  variance  forecasts. 

Recommendation:  Determine  a  more  accurate  method  of  calculating 
the  schedule  variance.  The  methodology  proposed  here  would  provide 
schedule  variance  forecasts  in  some  time  unit,  probably  days,  but  if  it 
cannot  be  implemented  perhaps  a  summary  event  network  of  critical 
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milestones  could  be  constructed  and  utted  to  provii  this  Information. 

An  alternative  is  to  enforce  the  spirit  of  the  C/SCSC  and  require  the 
contractor  to  work  the  variance  estimates  up  from  the  work  package 
level.  .(However  we  feel  that  the  effort  involved  in  accomplishing  this 
task  monthly  would  not  be  justified  by  the  information  obtained).  The 
estimate  reliability  and  the  workload  Involved  could  be  increased  and 
reduced,  respectively,  by  moving  the  forecast  date  from  "At  Completion" 
to,  say,  "Six  Month  Forecast". 

5.  Conclusion:  Of  the  two  regression  models  attempted  in  this 
study,  the  Gompertz  model  appears  to  give  significantly  more  plausible 
variance  forecasts  than  does  the  current  CPR.  However,  due  to  the 
inaccuracies  inherent  in  such  models,  heavy  reliance  on  them  would  be 
inappropriate. 

Recommendation:  The  Program  Office  should  follow  up  on  the  Gompertz 

model  to  assess. its  accuracy  in  predicting  future  cost  and  schedule 

variances  (in  dollars)  during  the  remainder  of  the  A-10  DT&E  program. 

% 

General  Conclusions  and  Recommendations 

1.  Conclusion:  Within  the  DOD  there  is  little  syntactical  conven¬ 
tion  with  regard  to  the  term  "risk  analysis".  In  talking  to  members  of 
other  program  offices  within  ASD,  we  found  that  "risk  analysis"  generally 
infers  analysis  of  what  we  have  called  "technical  uncertainty". 

Recommendation :  The  definitions  concerning  risk  analysis,  risk 
management,  and  risk  assessment  listed  in  the  AF  Academy  Study  should  be 
adopted  and  Incorporated  in  an  appropriate  00D  document. 

2.  Conclusion:  We  question  the  requirement  for  all  major  develop¬ 
ment  programs  to  engage  in  "formal  risk  analysis".  We  doubt  that  "formal" 
risk  management,  as  required  by  DO D  directives,  is  appropriate  in  programs 
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like  the  A- 10.  Compared  with  other  development  programs  like  the  B-l 
advanced  manned  bomber  and  the  F-15  air  superiority  fighter,  technical 
uncertainty  in  the  A-10  program  is  low.  This  is  primarily  due  to  the 
previously  completed  competitive  prototype  program,  where  two  prototype 
aircraft  were  built  and  tested  in  an  operational-like  setting  prior 
to  source  selection  for  the  DT&E  contract.  The  design  to  cost  criteria 
enforced  engineering  tradeoffs  that  further  reduced  the  "doctrine  of 
quality"  environment  often  blamed  as  a  major  cause  of  contract  perform¬ 
ance  degradation.  This  is  not  to  say  that  problems  in  the  A-10  program 
have  not  or  will  not  occur,  but,  in  our  opinion,  the  program  does  not 
warrant  a  costly  formal  risk  assessment  like  that  suggested  by  the  AFA 
Study. 

Recommendation;  Delete  the  requirement  for  all  major  development 
programs  to  employ  risk  analysis.  Prior  to  the  RFP,  the  SPO  should 
determine,  based  on  a  preliminary  assessment  of  program  risks  and  the 
intended  management  method,  whether,  when,  how,  and  to  what  extent 
risk  analysis  should  be  employed  in  the  program.  This  could  be  conducted 
under  some  broad  DOD  guideline,  but  the  actual  data  requirements  for  the 
specific  analysis  must  be  included  in  the  RFP. 

3*  Conclusion:  Regardless  of  the  sophistication  and  elaboration 
of  any  scientific  methodology  used  to  predict  program  performance, 
process  uncertainty  can  invalidate  the  model  and  the  forecast.  The  A-10 
program  is  a  case  in  point.  Pending  Congressional  funding  alterations 
may  delay  production  by  as  much  as  12  months,  with  the  associated  rise 
in  costs.  Externalities  like  this  are  difficult,  if  not  impossible,  to 
model  in  an  objective  manner. 


Recommendation ;  None 
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Suggestions  for  Further  Study 

The  weapons  acquisition  process  is  an  intriguing  yet  exasperating 
area  for  a  formal  research.  We  feel  that  further  study  could  produce 
measureable  benefits  for  the  DOD  and,  ultimately,  for  the  taxpayers. 

We  list  here  a  few  suggestions  for  such  future  efforts. 

SA1MS  Schedule  Variance.  Perhaps ,  with  some  alterations  to  the 
present  system,  more  accurate  calendar  time  variance  can  be  predicted 
and  reported  in  the  Cost  Performance  Report.  A  study  should  be  conducted 
to  determine  a  practical  means  of  doing  this. 

Technical  Performance  Measure.  To  exclude  the  performance  para¬ 
meter  in  most  R&D  programs  would  be  to  invalidate  the  assessment.  We 
feel  that  technical  uncertainty  "drives"  cost  and  schedule  uncertainties 
to  a  large  extent.  Pi  lently,  the  performance  variable  is  the  most 
difficult  to  address  since  it  is  not  homogeneous.  Thrust,  weight,  and 
turning  radius  pannot  be  aggregated  into  a  single  "performance"  factor. 

A  study  should  be  conducted  to  develop  a  transformation  algorithm  which 
would  yield  some  type  of  reasonable  performance  utile  for  aggregating 
these  heterogeneous  performance  elements.  Parametric  cost  estimation 
would  be  a  logical  starting  place  for  such  a  study. 

Formal  Risk  Analysis.  An  attempt  should  be  made  to  implement  a 
formal  risk  analysis  of  the  type  envisioned  in  this  study.  A  develop¬ 
ment  program  with  high  technical  uncertainty  should  be  chosen,  and  the 
effort  should  begin  at  the  conceptual  stage,  prior  to  the  Request  for 
Proposals.  It  should  continue  as  long  as  significant  program  uncertain¬ 
ties  still  exist. 

Sunmary 

Our  ultimate  purpose  in  writing  this  thesis  has  been  to  document 
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an  attempted  application  of  quantitative  risk  assessment  in  an  on-going 
weapons  acquisition  program.  We  hope  that  through  our  observations  and 
inferences  the  reader  will  gain  a  better  understanding  of  risk  assess¬ 
ment  and;  risk  analysis,  and  their  application  in  the  R&D  environment. 
Perhaps  these  conclusions  and  recommendations  will  provide  some  insight 
and  guidance  for  future  researchers  who  may  tread  similar  paths. 
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A-10  Program  History 
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Considering  tlraa  constraints  ok  this  study  and  the  lack  of  a 
convenient  manner  for  dealing  with  the  non-additive  performance 
variables!  the  Fairchild  Republic  Company  (PRC)  A-1Q  specialised 
close  air  support  aircraft  program  was  considered  a  near  ideal  test 
case.  From  the  outset  of  the  study  members  of  the  Program  Control 
Division!  A- 10  System  Program  Office  (SPO)  indicated  a  keen  interest 
and  a  willingness  to  assist  in  any  way  possible « 

The  A-10  program  la  unique  In  recent  weapons  acquisition  annals. 
Studios  on  the  need  for  such  an  aircraft  began  in  1967  and  culminated 
in  Development  Concept  Paper  (DCP)  23A,  which  was  released  6  April 
1970  (16).  That  DCP  outlined  the  need  for  an  A-X  (designation  for  an 
Attack  aircraft  in  the  conceptual  phase).  Principal  requirements  for 
tha  specialised  close  air  support  aircraft  were!  (a)  an  integral  anti¬ 
tank  cannon,  (b)  high  payload  capacity,  (c)  small  turn  radius,  and  (d) 
long  loiter  time. 

A  Request  for  Proposals  (RFP)  was  iosued  7  May  1970  and  six 

a 

companies  responded.  Two  companies,  Fairchild  Industries,  Fairchild 
Republic  Division,  and  Northrop  Corporation,  were  chosen  to  enter  the 
Competitive  Prototype  Phase  (CPP) .  The  prototyping  approach  had  not 
been  used  on  an  aircraft  development  program  since  the  ill  fated  North 
American  Aviation  B-70  effort.  The  CPP  required  each  contractor  to 
build  two  prototype  aircraft  and  submit  them  to  an  intense  operational 
prototype  competitive  flyoff  (31:46). 

Firm  fixed  price  contracts  were  let,  FRC  for  $41.2  million  and 
Northrop  for  $28.9  million  (7:14).  Authorization  for  these  contracts 
came  from  Defense  Systems  Acquisition  and  Review  Council  (DSARC)  I. 
Implicit  in  the  DSARC  approval  was  another  salient  feature  of  the 
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program  «  a  coat  bogey  of  $1.4  million  recurring  flyaway  coat  per  unit 
In  1970  dollar*  baaed  on  a  buy  of  600  aircraft  at  20  per  month.  Tha 
phrase  "design  to  coat"  chua  was  applied  to  the  program.  A  Buaineaa 
Week  article  headlined  "  'Design  to  cost'  la  the  Pentagon's  newest 
lues-phrase". 

The  now  Pentagon  policy  is  having  its  sharpest  test,  however, 
in  the  final  competition  between  Northrop  Corp.,  and  Fairchild 
Industries,  Inc.,  for  a  contract  to  produce  600  close  air  support 
aircraft-  Air  Force  Secretary  Robert  C.  Seamans,  Jr.,  has  ruled 
that  the  average  unit  production  cost  of  the  plane  (designated 
the  A-X  for  attack  experimental)  must  not  exceed  $1.4  million. 

It  it  does,  ha  warns,  there  will  be  no  Air  Force  purchasing 
orders  (lt4u). 

Subsequent  to  the  CPP  flyoff ,  FRO  was  chosen  to  receive  a  cost  plus 
incentive  fee  (CPIF)  contract  to  produce  10  development,  test,  and 
evaluation  (DT&E)  aircraft.  Secretary  Seamans  was  the  source  selection 
authority  and  used  a  weighted  system  to  make  the  selection  (40:14-15). 

The  choice  coincided  with  the  USAF  Source  Selection  Advisory  Committee's 
(SSAC)  recommendation.  The  FRC  prototype  used  General  Electric  (GE) 

TF-34  engines  identical  to  ones  used  in  the  U.S.  Navy  S3A  antisubmarine 
aircraft  program  but  the  DT&E  proposal  included  some  engine  modifications 
so  two  separate  Government.  Furnished  Equipment  (GFE)  contracts  were  let 
to  G.E. 

The  FRC  contract  was  let  at  a  nominal  $159,279  million  with  a 
70/30  government /FRC  share  ratio  for  costs  over  the  negotiated  price. 

Two  fixed  price  plus  incentive  fee  (FPIF)  contracts  were  let  to  GE,  one 
at  $27,666  million  for  the  full  scale  development  of  32  engines,  and 
the  other  at  $14,892  million  for  the  qualification  program.  A  smaller 
competitive  prototype  program  was  conducted  to  name  the  30mm  anti-tank 
gun  manufacturer.  The  competitors  were  General  Electric  Co.,  and  Philco- 
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Ford  Corp.  The  Philco-Ford  design  failed  to  perform  and  GE  was  given 
a  third  CFE  FPIF  contract  at  $23,754  million  for  the  gun  development. 

The  DSARC  confirmed  that  FRC  was  the  CPP  winner  on  17  January  1973. 
Within  a  week  executives  of  AVCO  Corporation,  builders  of  the  engine 
for  the  losing  Northrop  design,  protested  the  award,  sAying  that 
pressure  by  Long  Island,  N.Y.,  (location  of  FRC)  businessmen  and  politi¬ 
cians  had  influenced  the  USAF  decision  (45:22).  USAF  countered  with  a 
statement  to  the  effect  that  the  AVCO  engine  did  not  have  the  thrust  to 
power  the  heavier  Fairchild  aircraft  and  Lt.  Gen.  James  Stewart,  Commander 
of  Aeronautical  Systems  Division  (the  develop  manager)  said  of  the  A-10 
choice: 

The  Fairchild  people  had  their  eyes  further  downstream  in  the 
company's  approach  to  a  prototype  model...  We  can  do  a  great 
deal  more  with  tha  "Fairchild  prototype  than  with  the  A-9A. 

The  prototype-to-production  transition  will  be  much  better  with 
the  A-10A  (41:17). 

Representative  George  D.  Mahon  (D-Tex)  attempted  to  delay  the 
signing  of  the  contract,  contending  the  Texas  built  Ling  Tempco  Vought 
A-7D  attack  aircraft  would  perform  the  close  air  support  role  envisioned 
by  the  Air  Force.  He  was  sidestepped  when  USAF  signed  the  contract  on 
1  March  1973,  even  while  two  Congressional  investigations  were  still 
in  progress  (one  by  Rep.  Mahon  and  another  by  Senator  Lowell  P.  Weicker, 
Republican  from  Connecticut,  home  of  AVCO  Corp.).  That  action  displeased 
Congress  and  further  anti-A-10  sentiment  erupted  (40:14-15). 

In  an  attempt  to  appease  Rep.  Mahon,  USAF  released  to  him  the  results 
of  a  classified  computer  simulation,  Sabre  Armor  Charlie,  which  showed 
the  A-10  to  be  a  superior  choice  over  the  A-7D  in  the  scenario  envisioned 
for  the  aircraft.  Another  result  of  the  dissention  was  a  Senate  Armed 
Services  Committee  proposal  advocating  a  competitive  flyoff  between  the 
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A-10  and  the  A-7.  USAF  opposed  this  as  both  unnecessary  and  as 

Infeasible  with  only  the  prototype  A-10  available.  The  alternative 

of  postponing  an  A-10  production  decision  until  after  a  DT&E  aircraft 

(built  on  production  tooling)  could  be  flown  against  the  A-7  was  deemed 

too  expensive  and  too  time  consuming.  An  August  1973  article  in 

Aviation  Week  and  Space  Technology  quoted  one  USAF  officer  as  prefering 

a  prototype  flyoff  (albeit  without  the  A-10*s  primary  weapon*  the  GAU-8 

30mm  gun)  rather  than  waiting  two  years  for  a  production  configured 

aircraft  to  emerge.  That  same  article  further  states: 

The  thinking  reflects  a  growing  opinion  in  USAF  development  circles 
that  programs  should  be  adequately  funded  or  cut  off  completely 
at  an  early  stage,  rather  than  the  start,  go  slow,  halt,  restart 
cost-escalating  tendency  now  prevalent.  Primary  reasons  for  the 
trend  are  constrained  Defense  Department  budget  requests  and 
subsequent  congressional  cuts  triggered  by  the  current  anti¬ 
military  sentiment  and  past  pentagon  management  bungles  (14:17). 

Threats  of  program  cancellation  by  Congress  forced  USAF  to  agree 

to  a  flyoff  (49:22-23).  Senate  rules  for  the  flyoff  were  extremely 

vague: 

This  flyoff  would  not  have  to  be  too  long  or  complex.  The  main 
point  of  the  flyoff  is  to  take  experienced  combat  pilots  and  let 
them  fly  both  airplanes,  the  A-10  prototype  and  the  A-7D,  and 
then  make  a  judgment  as  to  which  airplane  they  would  rather  fly 
in  combat  (49:22-23). 

Senator  Harold  Huges  (D-Iowa) ,  author  of  the  above  quote,  had  sponsored 
a  bill  to  terminate  the  A-10  contract  and  the  flyoff  agreement  helped 
to  defeat  that  proposal  (49:14). 

From  late  September  1973  to  this  writing,  the  status  of  funding 
for  the  A-10  program  has  vascillated  through  various  Congressional 
conmittees.  The  most  recent  activity  was  to  restore  $10  million  cut 
by  the  Senate  and  to  sustain  a  $35  million  cut  in  long  lead  time  money, 
leaving  tha  program  at  about  $107  million  and  subject  to  production 
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delays  should  that  decision  be  made* 

The  impact  of  such  a  loss  would  probably  be  a  reduction  from  10 
to  6  DT&E  aircraft  and  up  to  12  months  delay  between  conclusion  of  the 
DT&E  pre-production  phase  and  first  flight  of  a  production  aircraft. 

The  flyoff  will  be  conducted  beginning  15  .ipril  197A  and  will 
use  Tactical  Air  Command  pilots  as  evaluators.  Since  there  is  not 
gun  in  the  number  1  prototype  aircraft  *  there  will  be  no  ordnance 
delivery.  Instead,  the  test  will  be  conducted  on  an  instrumented  range 
at  Ft.  Riley,  Kansas,  and  telemetry  data  will  be  used  in  reaching  a 
conclusion. 
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APPENDIX  B 

Properties  of  the  Triangular  Distribution 
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Surprisingly,  we  found  very  little  infomation  on  the  non- 
symetrical  triangular  distribution.  Although  the  VERT  simulation 
routine  is  programmed  to  handle  this  distribution,  we  thought  it 
important  to  derive  the  properties  of  the  triangular  distribution 
ino  order  to  obtain  a  working  knowledge  of  its  use:  • 


f(x) 


Figure  B-l  Triangular  Distribution 
.  where  xQ=  optimistic  time 


Standardizing  *-o  zero: 


x •  most  likely  time 
m 

xp“  pessimistic  time 


b  ■  x-x 
m  p 

C  “  x-x 
O  P 


The  area  of  the  triangle  ■  hc/2  «=  1  since  the  area  under 
the  pdf  must  equal  one.  Solving  for  the  height  (h): 

h  *  2/c 

Due  to  the  discontinuity,  the  pdf  is  given  by  two  functions: 

i£,(x)  -  2x/bc  ,  xs  xex^ 
x  pm 

f,(x)  “  2(c-x)/c(c-b)  ,  X  i  XSrX 
i  mo 


GSA/SM/74-1 


r 

The  expected  value  is  given  by:  lx f(x)dx  ,  therefore: 

E(x)  =  J 2x2/bc  dx  +  J 2(c-x)x/c(c-b)  dx 


c+b/3 


2  2 

The  variance  is  given  by  V(x)  =  E(x  )  -  [E(x)]  with  the 
following  results: 

V(x)  =  c2-3cb+b2/18 

The  cummulative  distribution  is  also  in  two  parts  with: 

2 

F,  £x)  =  x  /be  ,  x  i  xSx 
1  pm 

F2(x)  =[1b(b-2c)x(2c-x)  ]/c(c-b)+b/c  ,  Xffl< 
In  order  to  araploy  Monte  Carlo  techniques,  the  inverse  re¬ 
lationship,  x  =  G_1(y),  must  be  formed: 

x  ■  'Jybc  +  x  ,  O^y-b/c 
P _ 

x  »  c  ->l  c2(l-y)  -  bc(l-y)  +  xp  ,  b/c£yf.l 


x£x 
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APPENDIX  C 
Description  of  VERT 
[Extracted  from  Bevelhymer  (10)  ] 
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A.  Description  of  VERT  Process 

VERT  is  a  network  tool  which  utilizes  simulation  as  a  means 
of  deriving  solutions.  It  has  an  extensive  array  of  logical  and  mathe¬ 
matical  features  which  makes  it  possible  to  analyze  complex  systems 
and  problems  in  a  less  inductive  manner  than  traditional  methods. 

When  using  this  tool,  the  user  can  expend  more  time  on  individual 
component  time,  cost,  and  performance  analysis  rather  than  developing 
the  interaction  among  components.  The  extensive  number  of  operands 
available  removes  the  inductive  h ’.adaches  from  modeling  component 
interaction.  These  operands  enable  the  user  to  explore  conditional 
nonlinear  multivariate  situations  which  defy  ready  mathematical  analysis. 
VERT  enables  the  user  to  create  a  fourth  dimension,  "risk,"  which 
is  used  as  a  common  measure  to  integrate  the  three  principal  dimensions 
of  time,  cost,  ,a.nd  performance.  Risk  is  the  endogenous  variable  being 

controlled  by  the  exogenous  variables  time,  cost,  and  performance. 

* 

VERT  has  two  parts.  Part  one  consists  of  constructing  a 
graphic  network  representation  of  the  project.  Pr’-t  two  consists 
of  analyzing  that  network  through  the  use  of  a  computer  program. 

Figure  C-l  is  an  example  graphical  network  representation 
depicting  elemental  activities,  events,  and  real  time  decisions. 

Real  time  in  this  context  has  the  following  connotation:  the  decisions 
made  within  this  mathematical  simulated  network  would  be  the  same  as 
those  the  manager  on  the  job  would  make,  giVen  thctime,  cost,  and 
performance  values  derived  by  the  network  for  each  of  the  various 
decision  alternatives  provided  to  be  the  same  as  those  encountered  In 
the  actual  project  development. 
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Figure  C-l .  Armored  Reconnaissance  Scout  Vehicle 
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In  the  VKRT  system,  project  activities  arc  represented  by  ares, 
and  events  or  milestones  are  represented  by  nodes,  The  area  and 

especially  the  nodes  are  used  to  create  the  real  tine  decision  capability* 

» 

Therefore,  the  flexibility  and  array  of  capabilities  structured  in  the 
nodes  and  arcs  become  a  very  critical  consideration  when  attempting  to 
model  an  unusual  decision  situation*  The  guideline  followed  in  design1- 

*  1 

ing  VBRT  is  that  of  trying  to  strike  a  balance  between  having  enough 

features  available  to  efficiently  model  any  decision  situation  versus 

'  * 

over  burdening  the  user  with  features  to  the  point  that  only  technicians 
can  cope  with  this  tool* 

While  pictorially  describing  the  project  In  terms  of  tho  VRRT 

operands,  numerical  values  for  time,  cost,  performance,  achievement 

and  event  probabilities  are  assigned  to  the  various  project  elements. 

% 

Procedures  useful  for  oliciting  data  hava  been  suggested  by  Dalkoy  (1970), 
Northrop  (1970),  and  Raiffa  (1970),  Tho  numerical  values  assigned  must 
be  measured  in  a  consistent  number  throughout  the  network.  Time  cannot 
be  specified  in  torus  of  weeks  in  one  eection  of  the  network  and  In  terms 
of  years  elsowhcre.  Likewise,  cost  must  bo  measured  in  Identical  units 
as  ten,  hundred,  or  thousand  dollars,  etc,,  throughout  the  network. 
Performance  can  be  expressed  in  terms  of  any  meaningful  index  such 
•s  horsepower)  weight,  reliability,  utllos,  return  on  investment, 
quality  appraisal,  systems  worth,  e^o, 

Time,  cost,  and  performance  for  each  activity  .con  be  jointly  or 
•inguJarly  modeled  as  a  functional  relationship  with  othor  node  or 
arc  time,  cost  and  performance  parameters  in  the  network  and  as  a 
stochastic  variable.  This  dual  capability  enables  modeling  tho  functional 
relationship  portion  of  a  regression  equation  among  key  parameters  in 
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the  network  anti  additionally  modeling  the  stoahaetie  residual,  V15UT 


has 

the  following  14  transformations  to  aid 

In  the  task  of  expressing 

functional  relationships  among 

the  key  parameters. 

U2jl 

’Transformation 

No. 

Trannformatien 

1 

9 

C^lee^C^hXj 

2 

W*  ' 

■% 

10 

C^SinCGjX^J^Xj 

3 

C1(X1  *  CjhXj. 

11 

^iCosfCjXpi^Xj 

4 

cl(x1  -  c2)*x2 

12 

CifAretanfCjX^JJ^tj 

3 

'C2 

VX1  >*2 

n 

X1  ~  C2*  C1X1*X2 

otherwise 

* 

♦ 

6 

14 

X1  ~  C2‘  CiVX2 

« 

otherwise 

7 

• 

C2X1 

C1(e  l)+*z 

WX2 

8 

C^log^C^)]^ 

represents  &  time,  cost,  or  performance  value  previously 
derived  within  the  network.  and  C2  are  Inputted  constants.  Is 
an  ordinary  multiplier  of  the  transformed  variable  whllo  C2  Is  used  to 
transform  Xj  to  X^.  *  % 

The  functional  modeling  available  in  VERT  will  onable  deriving 
time,  cost,  and  performance  values  for  each  activity  as  a  function  of 
the  followings  (l.o.,  X^  can  be  any  of  the  following  previously  derived 
values)  (1)  node  (event)  timo,  cost,  performance  values  (2)  are 
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(activity)  time,  cont,  performance  values.  (A  parameter  must 
not  bo  dependent  upon  Itself  and  there  must  be  a  dependency  hior- 
archy  established  among  those  three  principal  parameters  if  time 
and/or  cost  and/or  performance  are  interrelated  for  a  given  activity.) 
To  aid  stochastic  modeling,  VERT  has  10  statistical  distribution  input 
options  which  arc  as  follows t  (1)  constant,  (2)  uniform,  (3)  normal, 

,  i 

(A)  triangular,  (3)  erlang,  (6)  lognormal,  (7)  poissoa,  (8)  gamma, 

(9)  beta— 3  or  A  parameters,  or  (10)  any  distribution,  entered  as  a 

»  • 

histogram  approximation  to  the  probability  density  function. 

The  degrea  or  extent  a  project  needs  to  bo  cogmonted  into 
activities  and  events  is  a  function  of  available  data  and  the  results 
desired.  Some  managers  prefer  to  estimate  parameters  for  entire 

modules  or  higher  level  work  packages,  rather  than  estimating 

* 

paramo Loro  for  the  smaller  elemental  Items  in  thoao  work  packages. 
Problem  site  sometimes  has  a  bearing  on  the  way  the  network  Is 
structured.  If  a  problem  is  large,  it  is  often  advisable  to  construct 
lower  levol  networks  (subnets)  of  major  modules.  The  histogram 
Inputting  capability  for  an  activity's  time,  cost,  and  performance 
enables  stochastic  substitution  of  results  from  lower  level  subnotworkn 
into  a  higher  level  network. 

Part  two  of  the  V&RT  procedure  consists  of  analysing  the 
network  through  the  use  of  the  computer  program  (Moeller,  1972). 
Networks  are  constructed  so  that  various  combinations  of  alternative 
activities  could  occur  to  tnaks  a  project  successful.  The  computer 
program  explores  alternate  ways  of  completing  the  project  through 
the  technique  of  simulation.  Upon  simulating  tho  network  a  sufficient 
number  of  times,  the  computer  program  prints  out  the  following  node 


88 


G8A/8M/74-1 
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time,  coat  (discounted,  if  desired),  and  performance  information! 

It  Pictorial  histogram  approximations  to  the  marginal 
probability  density  functions. 

'  ;  2.  Pictorial  histograms  approximations  to  the  marginal 
cumulative  donsity  functions  (see  Figure  F-l,  cell  data  are  printed  on 
the  page  following  tho  histogrum  printouts), 

3,  Mean  observations. 

4.  Standard  deviations, 

•  * 

3.  Coefficients  of  variation. 

This  information  is  displayed  for  all  internal  nodes,  intervals 

between  nodes,  and  terminal  nodes  as  requested.  In  addition,  all 
% 

terminal  node  time,  cost,  and  performance  data  aro  combined  to  give  a 

composite  terminal  node  time,  coat,  and  performance  printout, 

* 

Tho  histogram  printout  of  tho  probability  density  function 
provides  a  picture  of  the  range  and  concentration  of  time,  cost, 
and  performance  values.  Probability  of  exceeding  certain  value 
levels  can  be  obtained  from  the  histogram  printout  of  the  cumulative 
density  function.  The  mean  indicates  the  center  of  the  distribution 
while  tho  standard  deviation  gives  an  Indication  of  the  overall  spread 
of  the  distribution.  Lastly,  the  coefficient  of  variation  enables  an 
Inference  to  be  made  on  the  spread  of  the  distribution  in  relation  to. 
its  mean. 

a 

VERT  prints  out  a  bar  graph  of  terminal  node  utilization 
(slmilur  to  Figura  C-2).  It  is  through  the  use  of  this  printout  that 
the  project  "risk"  can  be  ascertained.  The  usual  form  a  decision  risk 
analysis  network  takes  is  that  of  having  one  or  several  terminal 
nodes  collect  successful  project  completions,  and  one  or  several 
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Figure  C  2  Armored  Reconnaissance  Scout  Vehicle 
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terminal  nodes  collect  unsuccessful  project  completions.  Realiza¬ 
tion  of  these  various  terminal  nodes  compared  to  the  total  number 
of  iterations  gives  an  indication  of  project  success  or  failure.  The 
program. next  prints  out  a  critical  path  index  for  nodes  (see  Figure  F-2) 
and  arcs  (similar  to  Figure  C-3).  Since  different  stochastic  paths  can 
be  realized  in  the  process  of  simulating  the  network,  the  critical 
path  tends  to  change.  Accordingly,  the  program  computes  the  pro¬ 
portion  of  time  each  arc  and  node  is  on  the  critical  path.  These  critical 
path  options  facilitate  making  sensitivity  and  crash  program  analysis. 

B.  Mechanics  of  the  VERT  Process 
1.  General  Processing  Steps 

The  processing  steps  of  this  program  are  highly  in¬ 
fluenced  by  the  various  states  the  arcs  assume  during  an  actual 
simulation  iteration.  These  states  are  as  follows:  '(1)  uninitiated, 

(2)  logically  eliminated — will  not  be  considered  as  a  feasible  activity 
for  this  iteration,  (3)  unsuccessfully  completed,  (4)  successfully, 
completed,  and  (5)  candidate  for  the  critical  path. 

An  approximate  sequence  (exceptions  discussed  in  Section 
I-B-2)  of  the  steps  the  program  takes  in  deriving  a  solution  for 
a  single  simulation  iteration  is  as  follows:  First,  all  initial  nodes 
are  processed  in  the  order  they  were  inputted  to  the  program.  The 
output  logic  of  these  nodes  selects  certain  output  arcs  for  processing 
and  the  remaining  outputs  arcs,  if  any,  are  given  a  status  of  logical 
elimination.  Whenever  an  output  arc  is  selected  for  processing,  it 
is  immediately  Monte  Carlo  processed  to  determine  its  success/ 
failure  status.  Time,  cost  and  performance  values  are  also  derived 
for  this  arc  via  the  functional  relationships  and/or  statistical  dlstrl- 


91 


GSA/SM/74-1 


92 


Figure  C-3.  Armored  Reconnaissance  Scout  Vehicle 
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button  Inputted  for  it.  Next,  all  the  remaining  nodes  are  reviewed  in 
the  order  they  were  inputted  into  the  program  and  all  candidate  nodes' 
(those  nodes  in  the  immediate  area  of  the  network  flow)  input  arcs 
are  checked  to  see  if  they  have  been  processed.  If  these  arcs  have 
been  processed;  time,  cost,  and  performance  values  for  these  nodes 
are  derived  via  the  input  logic  rules  discussed  in  the  next  section  of 
this  paper  (Section  I-B-2),  The  output  logic  for  this  node  selects  arcs 
for  processing  and/or  gives  them  a  scatus  of  logical  elimination.  All 
nonprocesscd  nodes  are  continually  reviewed  until  they  all  have  been 
processed.  The  optimum  terminal  node  is  next  determined  as  the  one 

with  the  shortest  completion  time;  lowest  cost,  or  highest  perform- 

% 

ance,  or  the  best  weighted  combination  of  these  three  principal  factors. 
VERT  provides  the  capability  to  partially  or  fully  cost  the  activities 
which  were  initiated  before  but  not  completed  by  the  time  the  optimum 
terminal  node  was  realized.  If  time,  cost  and  performance  data  dis¬ 
plays  for  internal  nodes  were  requested,  the  program  now  stores  the 
necessary  items  needed  to  complete  these  displays.  The  critical  path 
is  next  determined  and  stored  as  the  path  with  the  longest  completion 
time,  highest  cost,  lowest  performance,  or  the  least  desirable  weighted 
combination  of  these  factors.  VERT  enables  optional  suppression  of 
critical  paths  originating  from  certain  terminal  nodes.  The  program 
then  continues  on  to  the  next  ieration  repeating  the  preceding  steps. 

To  Increase  the  simulation  processing  speed  as  much  as 

1  *• 

possible,  nodes  should  be  Inputted  into  the  program  so  that  any 
given  node  causes  the  processing  of  input  arcs  to  nodes  Inputted 
after  the  given  node.  If  this  task  is  successfully  accomplished, 
the  program  will  need  to  review  all  the  nodes  only  once  for  e«efc 
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simulation  iteration. 

2.  Operands 

The  basic  building  blocks  (operands)  of  VERT  are 
nodes  and  arcs.  They  are  the  vehicles  used  to.  express  the  unique 
aspects  of  a  project.  Their  functional  relationships  are  so  inter¬ 
dependent  that  it  is  nearly  impossible  to  describe  the  functions  of 
one  without  describing  some  aspects  of  the  other.  Arcs  perform 
two  tasks  in  the  network.  Their  primary  task  is  that  of  representing 
project  activities,  and  their  secondary  task  is  that  of  performing  a 
logic  function  within  the  network.  When  an  arc  is  used  in  this  latter 
capacity  only,  it  is  referred  to  as  a  transportation  arc.  Every  arc 
in  the  VERT  system  is  characterized  by  the  following: 

a.  An  arc  name 

b.  The  name  of  its  input  node 

c.  The  name  of  its  output  node 

d.  Probability  of  arc  completion 

.  Transportation  arcs  require  specification  of  only  the 

preceding  four  attributes,  while  arcs  representing  actual  activities 
require  some  of  the  following  items: 

e.  Separate  equations  (structured  via  the  trans¬ 
formations  built  in  VERT)  for  activity  time,  cost,  and  performance.  . 

f.  Stochastic  variates  for  time,  cost,  and  per¬ 


formance. 

Nodes  having  Filter  #1,02,  or  #3,  and  time/cost/ 
performance  probability  output  logic,  which  are  later  discussed, 
require  output  arcs  to  carry  the  following  additional  Information: 

g.  Filter  9 1  -  upper  and  lower  limits  on  time 
and/or  cost  and/or  performance. 
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I 

N 

I 

T 

h.  Filter  0 2  -  upper  and  lower  limits  on  the 
number  of  successfully  completed  input  arcs. 

i.  Filter  #3  -  names  of  other  arcs  accompanied 

by  an' indicator. 

j.  Time/cost/performance  output  logic-probability 
distribution (s)  possibly  requiring  time/cost/performance  boundaries. 

There  are.  four  basic  input  logics  available  for  the  split- 
logic  nodes.  These  logics  are  defined  as  follows: 

a.  |  I  |  |  ^  "Initial"  input  logic  is  used  to 

-  start  the  network.  Multiple 

Initial  nodes  may  be  utilized  in 
a  network.  Time,  cost,  and  performance  values  assigned  are  zero. 

If  the  input  logic  for  the  following  nodes  is  not  satis¬ 
fied,  all  output  arcs  will  be  logically  eliminated. 

"And"  input  logic  requires  all 
input  arcs  to  be  successfully 
completed  before  the  network 

flow  can  continue  through  this  u.v  :  •>.«■  time  value  assigned  to 

this  node  is  the  maximum  path  •  *-i>  of  all  the.  input  arcs.  Cost 
and  performance  values  assigned  to  this  node  are  computed  as  the 
sum  of  all  the  respective  path  costs  and  performances  of  each  input  * 
arc. 

"Partial  and"  input  logic  requires 
all  input  arcs  to  be  successfully 
completed  or  logically  eliminated 
from  the  network.  If  at  least  one  input  arc  has  been  successfully 
completed,  network  flow  will  be  allowed  to  continue  through  this 


b. 


c. 

P 

A 

N 

D 
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node.  The  time  value  assigned  to  this  node  is  the  maximum  path 
time  of  all  the  successfully  completed  input  arcs.  Cost  and  per¬ 
formance  values  assigned  to  this  node  are  computed  as  the  sum  of 
all  the  respective  path  costs  and  performances  of  each  of  the  suc¬ 
cessfully  completed  input  arcs. 


"Or"  input  logic  requires  all  input 

arcs  to  be  successfully  completed 

or  logically  eliminated  from  the 

network.  If  at  least  one  input  arc  has  been  successfully  completed, 

network  flow  will  be  allowed  to  continue  through  this  node.  The  time 

and  performance  values  assigned  to  this  node  are  the  path  time  and 

performance  values  carried  by  the  input  arc  having  the  minimum 

path  time.  The  sura  of  all  the  path  costs  of  each  of  the  successfully 

completed  input  arcs  is  the  cost  value  assigned  to  this  node.  Arcs 

flowing  directly  and  indirectly  into  an  OR  input  logic  node  having 

Input  node  completion  .times  greater  than  the  completion  time  of 

this  OR  node  will  be  pruned  from  the  network  via  being  given  a 

status  of  logical  elimination  from  the  network.  If  an  arc  is  pruned 

from  the  network  in  this  fashion,  the  network  flow  will  be  restarted 

from  this  pruned  arc.  Restarting  tha  network  flow  will  reprocess 

arcs  branching  away  from  the  flow  into  this  OR  node  which  have 

been  erroneously  processed  as  a  result  of  this  unfortunate  pro- 

* 

cessing  position  of  the  OR  input  logic  node. 

1  '• 

The  following  six  output  logics  available  for  split 
logic  nodes  will  be  utilized  only  when  the  input  logic  can  be 
successfully  executed. 
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arcs  emanating  from  this  node. 


"Terminal"  -  This  logic  is  used  to 
end  the  network. 

"All"  -  This  logic  will  simultaneously 
begin  the  processing  of  all  output 

"Time  Probabilistic" 


"Cost  Probabilistic" 


"Performance  Probabilistic" 


Each  of  the  three  preceding  logics  will  start  processing  only  one 
output  arc.  Arc  processing  is  accomplished  on  a  probabilistic 
Monte  Carlo  basis  and  can  inclu’''  ~  time/cost/performance  consid¬ 
eration  if  desired.  The  probability-time/cost/performance  dependency 
situation  enables  inputting  three  different  sets  of  output  probabil¬ 
ities  of  process  initiation  separated  by  two  time/cost/performance 

boundaries.  These  boundaries  create  three  regions  where  the  three 

# 

probability  sets  apply.  If  the  time/cost/porformance  computed  for 

'  ■» 

the  node  lies  between  zero  and  tlme/cost/performance  boundary  one, 
the  appropriate  time/cos t/performance  domain  is  region  1.  Prob¬ 
ability  set  number  1  will  be  utilized  in  this  case.  Likewise,  if  the 
node  tlme/cost/performance  lies  between  tlme/cost/performance 
boundaries  1  and  2,  the  appropriate  time/cost/pcrformanco  domain 
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is  region  2  and  probability  act  number  2  will  bo  utilised,  Lastly,  if 

tho  nodo  tima/cost/porformance  lies  beyond  the  time/coat/porformanco 

boundary  number  2,  the  appropriate  time/coat/performaneo  domain  is 

region  3;  probability  set  number  3  will  bo  utilised. 

If  timu/cwat/porformance  conditioning  la  not  required, 

only  probability  sat  01  needs  to  be  specified  (any  of  the  precoding 

three  nodes  [c.,  d,,  or  e]  can  bo  utilised  in  this  particular  case). 

Likewise,  if  it  is  deemed  that  two  probability  sots  separated  by  one 

time/cost/performanco  boundary  fit  the  situation,  a  single  time/ 

coat/performance  boundary  point  and  probability  sots  01  and  02  need 

to  be  inputted.  The  probability-time/cost/porformnnce  dependency 
% 

capability  is  utilized  in  situations  whore  tho  chances  of  cartaln  activ¬ 
ities  being  initiated  depends  upon  the  timo/cot t/porformanco  realised 
at  key  milestones  within  the  network. 


terla  for  process  initiating  output  arcs  is  based  on  jointly  or  singu¬ 
larly  satisfying  time,  cost,  and  performance  constraints  placed  on 
arcs  emanating  from  this  node.  These  constraints  consist  of  upper 
and  lower  time  and/or  cost  and/or  performance  boundaries.  If  the 
node  time  and/or  cost  and/or  performance  lies  within  the  constraints 

0 

placed  on  a  given  output  arc,  that  arc  will  fje  initiated  for  processing. 
Otherwise,  the  arc  will  be  logically  eliminated  from  the  network.  N-l 
of  the  N  output  arcs  must  have  constraints  placed  on  them.  The  Nth 
output  arc  must  be  entirely  free  of  any  constraints.  This  arc  functions 
as  an  escape  arc  in  the  event  the  constraints  of  tho  other  output  arcs 


f.  F 

•»-  L 
T 

1 


"Filter  01"  output  logic  will  initiats 
the  processing  of  one  or  a  multiple 
number  of  output  arcs.  The  cri- 
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havu  been  violated,  Tht  escape  arc  will  be  logically  eliminated  from 

the  network  if  at  leaet  one  cenitrsined  output  arc  was  proeecacd, 

Boundaries  for  the  constrained  output  Area  can  be  (1)  overlapping , 

(8)  continuous,  or  <3)  nencontlnuoua,  l.e,,  having  gaps,  Thia  node 

can  be  prooeaaod  with  one,  two,  or  three  eonatrainta  simultaneously 

being  employed.  Moat  large-scale  projects  have  time,  coat,  and 

performance  conscruinija  which  ahould  be  obaervod,  It  la  appropriate 

to  uoo  thia  logic  to  filter  off  thoao  circulation  liberations  which  do 

'  » 

not  fall  within  tho  limits  of  the  time  and/or  coat  and/or  performance 
eonatrainta. 


1* 


nmn 


"Filter  12"  output  logic  ia  nearly 


T  the  aarco  aa  FIT  1  oxcopt  for  the 

following  foe tor a i  (1)  Only  one 

» 

constraint  rather  than  one  to  three  constraints  can  bo  placed  on  the 
output  area.  This  constraint  consists  of  en  upper  aud  lower  bounds 
on  ths  number  of  input  arcs  realised  by  this  node;  and  (2)  only  PAND 
input  logic  may  be  employed  with  FIT  2  output  logic,  FLT  2  output 
logic  is  useful  in  constructing  tosting  situations. 

h,|  |  F|  "Filter  #3"  output  logic  will  Initiate 

for  processing  one  or  a  multiple 
number  of  output  arcs.  The  cri¬ 
terion  for  process  initiation  of  e, given  output  arc  is  based  on  jointly 
satisfying  all  the  arc  completion  constraints  plowed  on  it.  Otherwlae, 
the  arc  will  be  logically  eliminated  from  the  network.  These  con¬ 
straints  are  specified  by  listing  the  names  of  arcs  functioning  as 


j*  * 

constraints  and  attaching  a  plus  or  a  minus  sign  to  those  names.  If 
a  plus  is  attached  to  the  arc  nmame,  thia  constraining  arc  must  have 
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boon  aueeeasfully  completed  before  the  output  ere  being  constrained 
oen  be  initiated.  If  e  minus  le  attached  to  the  ere  name,  thle  eon- 
etrelnlng  ere  nu at  have  been  unaueeeaafuUy  proceeded  or  logically 
eliminated  from  the  network  before  the  output  are  being  oonetrelned 
can  be  initiated.  H-l  of  the  H  output  area  to  thla  node  muat  have 
at  least  1  and  can  have  up  to  IS  are  completion  constraints  placed 

i  » 

on  them.  The  Nth  output  are  la  an  eaeape  are  which  will  be  initiated 
only  in  the  event  the  other  N-l  area  fail  to  be  processed.  The  eaeape 
are  will  be  logically  eliminated  from  the  network  if  one  or  more  oon- 
•r rained  output  area  are  prooeaaed.  Thla  output  logic  la  eapacially 
uaoful  for  qituatione  whore  successful  completion  of  prior  aotlvitiea 
or  the  failure  of  prior  activities  requires  the  initiation  of  other  actlv~ 

ltloe  positioned  farther  on  in  the  network.  * 

% 

For  the  preceding  split  logic  nodes,  the  path  time,  cost, 
snd  performance  values  assigned  to  tho  output  arcs  consists  of  the 
eum  of  the  Individual  time,  cost,  and  performance  values  derived 
for  those  activities  plus  the  tins,  cost,  snd  performance  values 

assigned  Co  the  arc's  input  node. 

« 

There  are  four  special  nodes  having  unit  logic  rather  than 
having  separate  input  and  output  logic.  They  require  an  indication  of 
how  many  output  arcs  arc  desired  to  be  processed.  This  number  is 
Indicated  in  actual  network  drawings  where  the  pound  sign  appeers 
in  tha  small  pictorials  accompanying  theso  definitions. 

a.  TCPLE-0  "Titiu  Cost  Performance  Link  Escape" 


node  has  N  input  arts  coupling  with 
ono  particular  output  arc. 


Additionally  there  must  bo  one  uncoupled  output  arc.  This  arc  plays 
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•  role  comparable  to  tha  voia  playad  by  tha  eaaape  area  In  the  previous¬ 
ly  dotinad  filter  output  logic. 

The  number  of  output  area  initiated  for  preceasing  da- 
panda  upon  how  many  input  area  were  aueeaaafuUy  completed  and 
on  how  many  output  area  were  deaired  to  be  processed.  One  or  all 

or  a  aubaat  of  all  the  linked  output  area  may  b«  initiated  for  prooaaa- 

\  ’ 

ing.  If  there  aro  more  aueeeaafully  completed  input  aroa  than  there 
are  output  arc  prooaaa  initiallaacion  requests,  the  followini  aeleotion 
logic  la  utilised.  Thoao  output  area  will  be  prooaaaod  whose  oorre- 
apondint  Input  arcs  form  an  optimal  subset ,  Optimal  x'ubaot  selection 
can  be  based  ,on  minimum  total  path  time,  cost,  or  maximum  path 
performance,  or  tho  Last  weighted  combination  of  these  three  factors* 

Tha  remaining  output  arcs  will  bs  logically  eliminated  from  the  network. 

« 

The  time  value  assigned  to  this  node  is  tha  maximum  time 

value  required  by  the  most  tins  consuming  are  in  the  optimum  input 

arc  subaot  if  time  is  used  as  tha  only  decision  criterion.  The  same 

pruning  logic  will  be  employed  by  this  nods  as  is  utilised  by  the  OR 

input  logic  uode  (see  last  paragraph  of  OR  logic  description).  If 

another  decision  criterion  is  usad  to  select  the  optimum  input  arc 

subset,  the  node  time  value  is  recorded  as  the  maximum  time  value 

of  all  input  arcs  which  havo  not  been  logically  eliminated  from  tho  , 

network.  The  cost  value  assigned  to  this  node  is  computed  as  the 

* 

sum  of  the  cost  values  of  all  input  arcs  that  have-not  been  logically 
eliminated  from  the  network.  The  performance  value  assigned  to 
this  node  is  computed  as  the  average  of  all  input  arcs  in  the  optimal 
input  arc  subset. 

If  the  output  arc  processing  request  cannot  be  fulfilled 
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entirely,  ell  output  ares  will  bo  logically  eliminated  from  the  net¬ 
work  except  tho  ecapo  arc  which  will  than  bo  procoeeed. 

Xf  tho  escape  avo  to  proceaaod,  tho  time  value  assigned 
to  thta  node  to  tho  maximum  time  value  of  all  Input  area  which  have 
not  boon  logically  eliminated  from  tho  network.  The  value  aaoignod 
to  node  performance  lo  ooro. 


b. 

I'TCPL 

R«#  "Partial  Tima  Coat  Performance 

[  Link  Escape"  logic  le  the  same  as 

nr 

j  ""  TCPLE  logic  except  that  the  number 

of  output  area  proceeding  request  need  not  bo  fulfilled  entirely.  If 
at  loaot  one, input  arc  hot  boon  successfully  completed t  tho  correa- 
ponding  output  arc  which  linka  with  thla  successfully  completed 

Input  arc  will  be  processed,  The  eocapo  arc  will  only  be  processed 

» 

when  all  input  area  to  thla  node  fail  to  be  successfully  completed. 


c. 

PLE-0 

"Preferred  Link  Escape"  has  the 

_ some  physical  makeup  aa  the  TCPLE 

« 

node.  The  only  difference  between 

these  two  nodes  io  the  logic  used  to  select  an  output  for  processing. 

Tho  logic  in  PLE  requires  that  tho  first  input  arc  be  given  preference 
over  tho  eocond  \nd  the  second  be  given  preference  over  the  third,  etc. 
Thus,  the  criterion  for  selection  is  preference,  not  time,  cost,  or  per¬ 
formance. 


d.  PPLE-0 


"Partial  Preferred  Link  Escape"  Logic 
is  analogous  to  PLE  as  PTCPLE  is 
analagous  to  TCPLE.  (The  number 


of  output  arcs  processing  request  need  not  be  fulfilled  entirely.) 
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For  the  preceding  special  nodes,  the  path  tine,  ooat, 
and  performance  value*  assigned  to  output  arcs  are  computed  as  the  sum 
of  tho  Individual  time,  cost,  and  performance  values  derivod  for  those 
arcs  plus'  the  path  time,  cost,  and  performance  values  derived  for  those 
linked  input  ares,  The  escape  arc  is  an  exception  to  this  rule,  Its 
time  and  coat  value  is  computed  as  the  sum  of  the  time  and  coat  values 

,  s 

derived  for  this  arc  p^us  the  time  and  cost  values  assigned  to  the 

input  nodo.  Path  performance  value  for  this  arc  is  compound  as  the 

individual  performance  value  derived  for  this  aro. 

These  four  nodes  aro  especially  useful  for  structuring 

major  command, or  board  of  director  type  decisions.  Since  these 

nodes  are  the  only  ones  which  can  accommodate  input  arcs  having 

probabilities  of  completion  less  then  one,. they  are  also  often  utilised 

« 

» 

as  network  flow  continuity  devices.  In  this  capacity  these  nodes  pre¬ 
vent  the  network  flow  from  dying  within  the  network. 


I 
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APPENDIX  D 

OMNITAB  Program  for  Least  Squares  Predictions 


/ 
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OMNI  TAR  MOIRAM  F OR  A -1#  COST  ACCOUNT  VARIANCE  REGRESSIONS 


US?  Or  COMMANDS*  DATA  AND  ClAONQSTiCS 
SCAN  72  t 

DIMENSION  SS  ROUS  X  140  COLUMNS 

HEAD  8  2 / SCHE  0  VAN 

HEAD  8 1/COS T  VAR 

HEAD  84/LA80R  NR  VAR 

HEAP  88/COEfr  8  SO'S 

HE  AO  89/RESIDUALS 

HEAD  90/PRED  VAL'S 

HEAD  71/HCuS 

HEAD  72/8CUP 

HEAD  71/ACTUAL  DOLRS 

HEAD  74/bUOO  NOUNS 

HEAD  7S/ACTUAL  HOURS 

HEAD  78/HONTHS 

HEAD  79/SCMEU  VAH'CE 

NEAD  80/COST  VAR'CE 

HEAD  81/LAH  VARIANCE 

HE  A  0  1/OESION 

HEAD  8/TOOL  PLAN'NO 

HEAD  tl/TOOL  DESION 

HEAD  16/TOOL  f  A  B  R  I  •  N 

HEAD  21/OUAL  ASSUR 

HE  A  0  26/PROCUREMfcMT 

HEAD  31/LOfTlNQ 

HEAD  16/REPROU'N 

HEAD  41/HATEHIEL 

HE  AO  4A/HAUERS  TOOL 

HEAD  51/HAOERS  OA 

HEAD  96/HAQERS  SUP'T 

HE  AO  61/SUHSYST  DESN 

HE  AO  7  t /UBS  HCUS 

HEAD  72/uBS  BCUP 

HEAD  71/UBS  ACTUAL 

HE  AO  74/UBS  SCH  LAB 

HEAD  75/UBS  ACT  LAB 

HEAD  135/8CMS  MAT-JUN 

HEAD  136/PKED  C S I  VAR  . 

HEAD  117/EARNED  VALUE 
HEAD  U9/8CUP»CST  VAR 
SET  78 

1  2  1  4  5  6  7  8  9  10  11  12  13  14  19  16  17  18  19  28 
SET  COL  1 

62800  101819  192794  167313  962132  980413  649974  19889  42029 
93032  69399  101SB8  118607  17B466  27719  46949  62479  84801 
110097  125934  159099  48341  76549  120138  146414  244967 
124798  391778 
SET  29  1 

19884  16405  53080  69119  116677  119705  166119 
SET  COL  7 

62999  99047  171018  294932  491699  964599  649812 
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ONNITAO  PROGRAM  FOR  A-10  COST  ACCOUNT  VARIANCE  REGRESSIONS 


LIST  OF  COMMANDS,  DATA  ANO  DIAGNOSTICS 

1669339  2444J65  2912869  3379170  3679731  4139239 

4139239  69094  173249  334972  638043  878231  1092379  1219748 

26740  41073  68423  81044  84268  67288  90394 

449166  729419  1027848  1331960  1664207  1894693  2039732 

3198  20099  28049  28049  28049  28049  28049 

SET  36  106 

121980  168629  21)678  274487  309416  329294  391377 
SET  36  109 

000  409820  409820  1024991  1024991 
SE1  112 

299641  347660  394009  380997  393887  406982  419961 
99087  7  7  2 II 9  80096  91638  96970  9884  1  101089 
32010  94067  66348  74428  77U93  79929  62213 
66023  97883  120691  146087  163836  176991  187094 
17716  31222  30240  42897  46411  49217  91461 
93922  117734  125497  136231  144090  151824  160803 
SET  36  115 

173595  196292  216639  242942  242942  242942  322463 
SET  36  110 

493390  979304  575304  975304  979304  575304  579304 
SET  121 

90671  90671  90671  90671  90671  90671  90671 
29172  90475  81821  103291  121342  133111  139838 
245208  269471  311581  332290  430285  453241  46U198 

390467  447108  450350  497049  471434  473869  476336 

306597  374274  3820*7  400610  911533  554807  554807 

43150  71550  109760  147260  172580  190080  193890 
SET  36  1.24 

38438  46471  93452  77906  94103  104940  107239 
SET  36  127 

106746  115933  123645  131758  134397  136670  139229 
SET  22  130 

•2189  94934  109965  125397  142054  142054  156711 

1/  HAUU  MATRIX  IN  1,1  SZ  42X9  TO  1,71  SZ  42X5  RUT  IN  1,71 

2/  INCREMENT  1  BY  8500000008 

PERFORM  1  THRU  2  13  TIMES 

her  page 

NOTE  . . . 

NOTE  H 8 S  TOTALS  ARE  IN  A  42X9  MATRIX  LOCATED  AT  1,71 

NO  IE  ••••••••••••••••••••••••••••••••#••••••••*••#•••*••• 

FIXED  0 

PRINT  71  •••  75 
NEM  PAOE 

2.1/HAUO  1,71  SZ  7X5  TO  43,71  PUT  IN  43,71 
2.2/  INCREMENT  2.1  BY  7  0  0  I  0  0  I  I 
PERFORM  2.1  THRU  2,2  »  6  TINES 

4/  NAOU  MAIRIX  IN  1.1  $Z  7X69  TO  43,1  SZ  7X69,  PUT  IN  43,1 
5/  INCREMENT  4BY706S00I6I0 
PERFORM  4  THRU  5,  6  TINES 

NO  IE  MM. . IMMIttMl . IMMMMMMMMMIIMM. 
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OHN 1 T  A ■  PROGRAM  FOR  A-10  COST  ACCOUNT  VARIANCE  REGRESSIONS 


LIST  Or  COMMANDS,  DATA  ANO  DIAGNOSTICS 

NOTE  FUNCTIONAL  TOTALS  ARE  STORED  IN  A  7X65  MATRIX  AT  43.1 

note  •••••••••••••••••••••••••••••• . •  •••••••••••  ••••!!  ••••••• 

SPACE  5 

5.1/  MAO 0  43.1  SZ  7X5  TO  50,71  PUT  IN  50.71 
5,7/  INCREMENT  5.1  9T  05000011 
HPHINT  43,1  SIZE  7X65 
NEW  P  A  OE 

NOTE  . . . . . . . . . ••••* 

NOTE  UBS  BASIC  STRUCTURE  TOTALS  ARE  IN  7X5  MATRIX  AT  43,71 

note  . . . 

SPACE  3 

NOTE  . . . . . . . * . . . 

NO  I E  FUNCTIONAL  BASIC  STRUCTURE  TOTALS  ARE  IN  7X5  MATRIX  AT  50,71 

NOTE  . . . 

SPACE  10 

MPRINT  43,71  SZ  7X5 
SPACE  5 

PERFORM  5.1  THRU  5.2  13  TIMES 

MPRINT  MATRIX  IN  50,71  SIZE  7X5  (FUNCTIONAL  TOTAL  VARIANCE  MATRIX) 

ASUBTRACT  1.71  SZ  42X1  MINUS  1.72  SZ  42X1  PUT  IN  1,79 

ASUB  TRACT  1,73  SZ  42X1  MINUS  1.72  SZ  42X1  PUT  IN  1.B0 

ASUBTRACT  1,75  SZ  42X1  MINUS  1,74  SZ  42X1  PUT  IN  1.G1 

7/  ASUBTRACT  43.1  SZ  7X1  MINUS  43,2  SZ  7X1  PUT  |N  50,1 

0/  ASUBTRACT  43.3  SZ  7X1  MINUS  43.2  SZ  7X1  PUT  IN  50,2 

9/  ASUBTRACT  43.5  SZ  7X1  MINUS  43.4  SZ  7X1  PUT  |N  50.3 

10/  Increment  7  by  0500050005 

11/  increment  b  by  0500090005 

12/  INCREMENT  9  BY  0500050005 

PERFORM  7  THRU  12  .  13  TIMES 

12.1/  A  A  UD  1,91  SZ  42X1  TO  1,133  PUT  IN  1.133 
12,2/INCHEMEnT  12.1  BY  03000000 
PERFORM  12.1  THRU  12.2  14  TIMES 
12.3/AAOl)  1,133  SZ  7X1  TO  50,133  PUT  IN  90.133 
12.4/INCHENENT  12,3  BY  70000000 
PERFORM  12.3  THRU  12,4  0  TIMES 

20/  HAPO  MATRIX  IN  1.79  SZ  7X3  TP  43.79  SIZE  7X3  PUT  IN  43,79 
30/  INCREMENT  29  BY  7000000100 
nen  paoe 

PERFORM  29  THRU  30  6 T I  ME S 

NOTE  . . . . . . . . . . 

NOTE  Mbs  BASIC  STRUCTURE  VARUNCES  ARE  IN  7X3  MATRIX  AT  43.70 

NOTE  . . . . . . . . . 

SPACE  3 

NOTE  . . . 

NOTE  FUNCTIONAL  BASIC  SIRUCtURE  VARIANCES  ARE  IN  7X3  AT  50,70 

NO  IE  . . . . 

SPACE  5 

MPRINT  43,79  7X3 

31/  MAUD  MATRIX  IN  50.1  SZ  7X3  TU  50,7*  SZ  7X3  PUT  IN  51.79 
32/  INCREMENI  318Y950VOOOOOO 
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OHHITAB  PROQRAM  TOR  A-19  COST  ACCOUNT  VARIANCE  REGRESSIONS 


UST  OF  CONNANOS,  OATA  AND  DIAGNOSTICS 

PERTORN  31  THRU  32  13  TINES 
'SPACE  It 
NPKINT  59, 79  7X3 
FLEXIBLE 
SPACE  3 
RESET  7 

14/  HHOVE  MATRIX  IN  1,79  SZ  7X3  TO  1,92 

19/  ST  I T  V  IN  COL  82,  UTS  1.0  TO  1  VAR  IN  COL  78,  COErr  88,  RS  89 

16,1/  AHOVE  1,88  SZ  1X1  TO  1,66 

16.2/  INCREMENT  16.1  8Y  0  9  0  9  0  1 

29/  Increment  19  by  i.  o.o,  o  9  0  9 

23/  PERFORM  19  THRU  20,  2  TIMES 

23.9/  INCREMENT  16.1  8Y  0  0  9  0  1  -3 

24/  increment  19  by  -3.  o.o.  o  o  o  o 
27/  increment  h  by  7  o  u  o  o  o 
PERFORM  14  THRU  27  ,  6  TIMES 
14/  HMOVE  MATRIX  IN  99,1,  SZ  7X3  TO  1.92 
16.1/  AMOVE  1,96  SZ  1X1  TO  97.1 
16.2/  INCREMENT  16.1  BY  U  9  8  9  1  9 
23.9/  INCREMENT  16.1  BY  U  9  9  9  -3  1 
27/  increment  14  by  o  s  t»  o  o  o 
PERFORM  14  THRU  27  ,  13  TIMES 

19/  FIT  Y  IN  COL  82  UTS  1.9  TO  1  VAR  IN  COL  78,  COEf F  88,  RES  99 

MMOVE  THE  MATRIX  IN  43,7V  SZ  7X3  PUT  |N  1.82 

16.1/  AMOVE  1.86  SZ  1X1  TO  7,66 

16.2/  INCREMENT  16.1  BY  0  9  V  9  9  1 

16.3/  DEFINE  THE  VALUE  IN  1,88  INTO  COL  99 

16,4/  MULTIPLY  THE  VALUES  IN  COL  99  79  AND  PUT  IN  COL  99 

18/  PAGE  PLOT  DATA  IN  COLS  82,90  AOAINST  DATA  IN  COL  7B 

18.2/  PLOT  82,90,76 

16.9/  INCREMENT  16.2  BY  1  I  9 

21/  INCREMENT  19  BY  1  9  « 

PERFORM  19  THRU  21  3  TIMES 

AMOVE  43,69  SZ  7X1  TO  1,136  (COST  VAR  -  MAY  THRU  NOV) 

AHOVE  43,71  SZ  7X1  TU  1,139  (BCWS  •  MAY  THRU  NOV) 

DEFINE  THE  VALUE  IN  7,67  INTU  COL  99  (TOTAL  COST  COEFF) 

SET  97  INOEPENUENT  VARIABLE  MONTHS  DEC  THRU  JUN 
8  9  19  11  12  13  14 

MULTIPLY  COL  87  BY  COL  89  PUT  IN  COL  87  (PRED  COST  VAR) 

AHOVE  1,87  SZ  7X1  TO  8,136  (PRED  COST  VAR  «  DEC  THRU  JUNE  > 

AMOVE  98,113  SZ  7X1  (0  6,139  (6CUS  MONTHS  DECEMBER  THRU  JUNE) 
SPACE  19 

NOTE  COL  139  >  BASIC  STRUCTURE  BCWS 

NOTE  COL  136  «  COST  VARIANCE  IN  RONS  W«PRED  COST  VAR  ROUS  8-14 
SPACE  9 

NPKINT  1,139  SZ  7X2 
SPACE  9 

NIUE  MATRIX  Or  COEFFICIENTS  FOR  FUNCTIONAL  AREAS 
NOTE  COLS  1*14  •  8,E,F,O,N,P,U,Z,2,6»7.6.C.T0TAL 
NO  I E  ROUS  97.J9  *  SCHEDULE  COEFF,  COST  COErr.  LABOR  COErF 
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OHNITAB  PROORAM  FOR  A*10  COST  ACCOUNT  VARIANCE  REGRESSIONS 


US!  Of  COMMANDS#  OATA  ANO  DIAGNOSTICS 
SPACE  » 

HPRINT  57.1  SZ  3  X 1 A 
SPACE  5 

NOTE  MATRIX  Of  COEfMCIENIS  FOR  MBS  ELEMENTS  OF  BASIC  STRUCTURE 
NOTE  COLS  66*66  ■  SCHEDULE  COEFF.  COST  COEFF.  LABOR  COETf 
NOTE  ROMS  1*7  *  0 1 9 , 0 4 9 . 0 59 . 0 69 , 0 7* . t »7 . T 0 T AL 
SPACE  5 

HPRINT  1,66  SZ  7X3 
RESET  7 

HEAD  63/EARNEO  VALUE 
MMOVE  43,71  SZ  7X2  TO  1.62 

FIT  Y  IN  63  UTS  1.0  TO  1  VAR  IN  62,  COEFF  116  ,  RES  IN  111 
RESET  14 

DEFINE  1,110  INTO  113 

ANULI  1,113  SZ  7X1  RV  6,135  PUT  IN  6,137  (PREO  EARNED  VALUE) 

AMOVE  43,72  SZ  7X1  TO  1,137  (ACTUAL  EARNED  VALUE.  MAV-NOV) 

ADD  137  TO  136  PUT  IN  139  (ACTUAL  AND  PREO  ACTUAL  COST.EV»COST  VAR) 

PLOT  135,137,139  VS  76  1 . 0 , 1 4 . 6 • 0 . 6 , 2 0 0 0 0 6 0 0 . 0  ( bCuS, 8CMP, ACMP  VS  T> 

PAGE  PLOT  135, 137, 139, VS  76,1.6,14.0,0.6,26066606.6 

PRINT  135,137,139 

LIST 

STOP 


NATIONAL  BUREAU  OF  STANDARDS,  M  ASH  1  NO  T  ON .  D.  C.  20234. 
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APPENDIX  E 

Gompertz  Regression  Program 
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02/28/73  LOGGED  IN  AT  13  .03.52. 

WITH  USER -ID  GK 
EQUIP/PORT  13/04 
COMMAND-  EDITOR. 

..F,T=,,li,21 
. .  C 

.100=2,  =  NUMBER  OF  DATA  SETS 
110  =  1  ,  =  FIRST  DATA  SET 

120=5,  = NUMBER  OF  DATA  POINTS  IN  FIRST  DATA  SET 
130=1. ,11.,  =  INDEP  VAR  AND  DEP  VAR 
140=2. ,21  . 

150=3  .,28.2 
160  =  4. ,42.1 
1  70=5.,  48. 

180=2,  = SECOND  DATA  SET 

190  =  4,  =  NUMBER  OF  DATA  POINTS  IN  SECOND  DATA  SET 
200  =  8. ,34. 2 
210=2  .3,9.1 
220=12.5,55. 

230=21  .1  ,86.2 

240  =  1  ,  =  NUMBER  OF  EXINT  POINTS 
250=15.5 
260  =  = 

.  .1  71  =2 , 

172=3.5,  =NUB  MBER  OF  EXINT 
173  =  7. 

L,A 


100=2 

= NUMBER 

110  =  1 

=FIRST 

120  =  5 

=  NUMBER 

130  =  1  . 

11  . 

140=2. 

21  . 

150=3. 

28.2 

160=4. 

42.1 

170=5. 

48. 

171=2 

» 

1  72=3  .5 

= NUMBER 

173  =  7. 

160=2 

sSECOND 

190  =  4 

= NUMBER 

200=8. 

34.2 

210=2,3 

9.1 

220=12,5 

55. 

= INDEP  VAR  AND  DEP  VAR 
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230  =  21.1  8$  .2 

.  240=1  =NUMBER  OF  EXINT  POINTS 

250  s 1 5  #5 

..1  71  =2,=NUMBER  OF  EXINT  POINTS  TN  FIRST  DATA  SET 
172=3.5,  =  IND  VAR  OF  EXINT  POINT 
L  1  71  1  72 

171=2  =NUMBER  OF  EXINT  POINTS  IN  FIRST  DATA  SET 

172=3.5  sXND  VAR  OF  EXINT  POINT 

•  •  SAVE,  TAPE7 

.. ATTACH, AA, GROCRV, CY=4 
ATTACH,  AA,GRCCRV,  CY=4. 

CYCLE  **,  GROCRV 
PFN  FOUND  IN  SD  001 
CYCLE  04,  GROCRV 
..CONNECT,  INPUT, OUTPUT 

•  .RU,  F,  F  =AA 
JOB  COMPILING. 

2.724  CP  SECONDS  COMPILATION  TIME 

TYPE  IN  NAME  AND  EXTENSION 

JOE  G  JONES  EXT  52549 


PROGRAM  TO  FIT  A  GROWTH  CURVE  TO  A  SET  OF  POINTS 
FOR  DATA  SET  1 

TYPE  V  IF  PEARL  IS  WANTED, TYPE  "o"  OTHERWISE  1 
JYPE  V  IF  GOMPERTZ  IS  WANTED, "0"  OTHERWISE  0 
TYPE  V  FOR  VON  BERTALANFFY,  "o"  OTHERWISE  1 
TYPE  *V  FOR  HYPERBOLIC  TANGENT,  “o"  OTHERWISE  0 
ENTER  UPPER  LIMIT  OF  GROWTH  CURVE  FOR  DATA  SET  (D)  60. 


0NA/8N/M-1 


tuTUM-  a : vi.  mtu  it  uu  i\:\ 

nOWS  v».  Qo»r«rt»  Pri-dlotid  1EW3  (J  *  125  million) 


rui-i?i  r in*  i  nv.Tin: 
un:r\r,v \  n:v  «  i»<'Ceeb&i2 
HCMHirr  urjfFiui:’*  »  - •  it 7 3 1 8* 

cf.;:mai''"i  n*  i- ;  *  rvvir;:  * 

cr.r:u,Min:  t'unu;;":  •  u?« 


Months  Millions  of  I 


IKU!  VAI 

1 .  G  0  G  0  C  0 

r.  ooccor- 

3 . 0  0  C’  C  0  D 
A.  PC 0  0  0'' 

r- .  o  ( i  •:  (.  c 

c .  r  c  o  c  o  o 
7 ,  c '  r  r.  r  c 
•  i  o  r  o  ?  *  o 
* .  o  o  c  o  c  c 
I c«  cococc 

11 .  c  c  o  o :  c 
ir.Goccco 

1 2 .  e  o  p  o :  o 

K .  0  C  P  0  0  0 

n.ocooc: 
go. ccoroc 
26. 000000 
32. OOOGOO 
31. OOOOCO 
A 1  .  0  0  0  0  C  c 
EMi  crccrv 


? r.r  i  tv.:  T 
0.».  17  32* 
f  .  !>3;:r’*? 
7.f 107 r: 
it.  -’.'c? 
!♦..  r; 

r  • .  C  t  7"  7  c 
fP.f C3303 
t  i  o'-rc 
22**6  11  .‘.7 
3n  .  0  0  *•'  6  6  V 
<W,.?7  1 &  7  3 
S0.7U  is*. 

C6C.7 
6  V .  0  L  f  A  C  C 
77. 3  Pi,*.*. 
■*r».«£P3S3 
107.0  C  r  7  A 
i  u  .ico'.o  in 
irc.po  ww.t 
ire. toes  12 
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u(m.M-  u:vi,  ip.tlx  t(  iau*  :m  i 

Pi'«diot«fi  Qp»t  Qurvo#  for  Buttle  Struoturea  (Eat.  •  31  Mp.W.6.8,  1*46,8) 


n.cru’ffir;)  luatk,:: 
crijr.’..::1:  »  i •  7 1 

riGMiMri:  c'.nur.i’T  « 
cmmur"  n>  l:  p-’rpv.vK::  « 
coniKATir;:  trEmcur;  *  -.*"*771:; 


Month* 


$  x  103 


i  >  *-  ^  i  \  4  %' 

1  ♦  0  0  0  c  c  c 

P. COOCOO 

3. CCC090 

4.  c  0  c  0 1  c 

5.  coot  to 
e .  0  0  c  c  0  0 
7. c coo Vo 
n .  0  0  c  0  c  0 
9 .  0  c  0  0  0  c 

1 0 . 0  c  e  0  0  c 

11. 0C0G00 

is. oc 000c 

13. OCOCCC 
IA. 0C00C0 
l 5.0  COO  CO 
l^.COOCCC 
si. 000000 
24. 0CCC00 
27. O00CCC 
30.000000 
33.CC000C 
36. Q000C0 
3«.  0  0  0  G  0  0 
EMI  crcciv 


k  :  e 

C.2,2  7 

/,{ 7 

.verm 

7  52 

• n  C 17  7  0 

IhiP 

.  M  F  p  4  ' 

PS  At 

.224107 

3207 

.  4'"'  0  4  0  ‘i 

5  6  01 

.9  6-5  Cl 

76 ’M 

.  It  1  1.  C  4 

l  00  0  4 

. ? 7 24 re 

|  r  1;  n  «% 

,  1  I'M  1 6 

15  271. 

.  n  3  C  0  l  3 

1  ’<  C  1  0 

.  26  24  2.5 

207  20 

.  r  1C  £2 

2320° 

.  542/'  24 

2  5rJ7; 

.  3  C  7  3  0  3 

f..  "2.43 

^  »*•  ^  f. 

•  X  C  :  O  *'• 

34216 

.0UU1 

3*537 

.477605 

4  1  4' 

.  20C4*<S 

4343  1 

,  5  2PC77 

44  C"  1 

.47  25  57 

45474 

. 65  57  EC 

459  7  2 

•  £  C  £7  C4 

4619  7 

.557113 

♦ 


Oop^  avd’cAlo  ^ 

ponait  fully  teslWa 
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cci  rt!  vv  u:.ve  nmi  u<  iiM/»  ut  i 

Fr#diot#d  Ooot  Ourvtt  for  Bmlo  Struotur##  (Kst.  #  33  Ko.»58.1»  L*53.1) 


:  ::o' ;  rr  irr 


h  r  t'i’.TIf 


»  * 
t  i 


cnm v.,-r 


I.709OI3&I 


i  i-g:e:  u? :■  icnt:v  ■ 
cntH’itin:v  cv  n:'A7if ::  «  . 
cc: : ccmicuNT  « 


r.n'it-  j 

£  3C 


Month# 

iNLEr  vv.r 

1 .  cocccc 

2. COOC-GC 

3.00  coco 
A. C COO  0  0 
5 .  C  C  0  C  0  C 
C. QCOOCQ 
7.  f?  C  0 C  £0 
«  .  0  0  C  C  0  0 
P.00000C 

I c. cococo 

I I.C000CC 

ir.occcco 


•  xio3 

K':  LUACT 
45  3.9A7C45 


3  7 r- 

514C 

3  9 

1  5  5  l 

4  39  7 

9  6 

£•5  32 

07  ir. 

C2 

39  C  1 

r  co 

15 

55  5  3 

3776 

r. 

7  5'V; 

‘  1 

u  t 

°9  45 

1  7  £  C 

75 

12  5  45 

r/.s* 

68 

15323 

5762 

8  '! 

rc  Ct¬ 

1 «:  1 8 

75 

rl  1  1G 

C  5-677  1 

13.0CCCC0  2  2S"C 7 7 
14.  0  0  0  0  DC  2679  1 .  £239 69 
15.0DCCC0  29457.217376 
18.000000  30.438  »  37766? 
81. 000000  41743. 287996 
24. CO CC 00  4EE 31. 131855 
27.000000  48 129.767593 
30.000000  49767.736061 
33.000000  31011.032747 
36. 00  00  CO  51755.420  105 
33.000000  5£C^7. 132539 


end  cr.ccr.v 


(Copy  bvc.llablo  *  * 

poittllt  iuily  Icgiblo 
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crm.:n-  ci:  vi  unu  it,  w.ta  m  i 

Predicted  Cost  Gurvss  for  D^slo  Struoturos  (Eat,  0)  38  Ho, *54*8,  1*56.0) 


rtdrfciiin:  uh’atio:’ 

CCJJSTA11T  TE!  r  *•  1.71516135 

rhorifcic:'  c./smcrsi'T  ■  i /, *. 

cmrici:  n?  l»  ti  :  mv-Tin:  ■  c  5 

cor:LL/,Mcr  cruncHi:!  ■  -.r^- 


Months 


$  x  10J 


T*. 

va: 

1. 

c  0  0  o  c  c 

45".  9  2".?  3° 

8. 

ccoooc 

94  c  1  ro 

3. 

ocooco 

1  5  5  1  •  4  0  7°  5-9 

4. 

ccoooc 

252  5 . 84"' -  3 2 

5. 

rococo 

3-47. 1 5 1 V39 

6. 

000000 

5 5 35.2*: 72  6  4 

7. 

coo  go;. 

7  5  ■  0  .  1 "  1  "  *:  2 

o 

1C 
1  1 
12 

13 

14 

15 
13 
81 
24 
27 
30 
33 
30 
33 

EML 


» c  c  o  o  c  s 
•  0  C  0  c  c  c 
.  0  0  0  0  c  c 
.rococo 
.  0  0  0  0  00 
,  0  0  0  0  0  0 
, 00CG0C 
,000000 
.000000 
.  oocooc 
.000000 
.000000 
.000000 
. 000000 
.  0  0  0  0  0  0 
.oocooc 
Grccr.v 


^946  . 5? : 544 
12573. 302744 

1  54  **‘.4  34 £43 
1-30 1 .440233 

2  1  3 0  5. 450 0  56 
34354. *01 677 
27872.326317 


2CC73 
374?-. 
43236 
47  39  7 
5  0  293 


.926443 
.296463 
.509331 
.?.*  4  33  6 
.  760474 


52255.907693 
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54423.2 67345 
54924. 4552 14 
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GPITLCT?.  Cl!  V1-.  riTTiL  TC  CAT/,  £L1  1 

Predicted  Cost  Qurves  for  Basio  Structures  (Eet.  i  38  Ho.*58.Cy  L*59.4) 


rccrrrcic::  L' vatic:’ 

CCUfTAIll  T IIV  *  I. 7259? £75 

rEo:  tsricr  crr.FFicuii?  » 

ccefi-’ i ci lkt  t:;*  lltl:ti;.y.tic;’  »  .99965 

CGI.r.LLATIC:’  CCCI'  FI  CI  i.VT  »  -  .  £  £  0 


Months 

IN CLC'  VAC. 

1 .  C  0  C  0  C  C 
2. OOOOCO 
3. COCOCC 
A. 00C00C 
5.  OCCOC'C 
6. OOCOOC 
7. 0GCC00 
8. 00000G 
9. OGOCCC 

10. C0CC0C 

1 1. CGG0G0 
12. 000GCG 
13. COCCOC 
14.000000 
18. OCGCOC 
18. COG  000 
2 1 .  0  0  C  0  0  C 
24. 0  C  0  C  0  0 
27. OOOOCO 
30.000000 
33. OOCGOO 
36.000000 
38 . 0  0  0  G  0  0 

LMD  GCCCrV 


#  X  103 

FC 'LCAJT 
467.5869 83 
■‘77.6801  04 
1 542.92032? 
281 6 . Cn33C7 
38  4  0 . 2  58  3  1 9 
8540. 09706? 
761 1.303099 
1002C. 664353 
1271 5. n0 7° 52 
15629 ,727241 
1 8 60 7. n 0349 5 
21016.467194 
249  46. 47 6  DEC 
c.  8  010.223  0  0  4 
3098  2. 1  C  35 
309 1 4.436 1 32 
45-126.726,672 
49  68  5.  6  55754 
52-092.  1  07797 
55095.762409 
5 6  5 5° . 735 1 44 
5753-8. 599  1  C3 
579^8 . C  07765 


<T!*r 

yorjidt  58 
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rr»-r !  r  it  cr:  vl  mttli.  to  lata  sti  i 

Predioted  Coat  Curves  for  Basic  Structures  (Est.  0>38  Mo.*  44.2,  L»44.75) 


rxdrUicici;  lciatici: 
cci:fta:*t  t t : ♦  ■  i.69no3i9 

I'ECrtiSIC-!:  CCLFi-  i  cult  ■  -.16007n57 

CnLI'HClLLT  Cf  Ll-Ti.:  m:  H  .99952 

COl'MLATICi:  CCHFFI  Cl  t!’T  =  -.9997606 


Months 

ir:LLr  vaf 
1. eooooc 
2. ococco 

3. OCCOCO 
^.COCOCO 
5. CCOOCC 

e.  o  g  c  o  o  o 

7. CCOOCC 

s .  0  c  c  c  c  c 

9.000000 
I0.CGCGCC 
1 1. OCCOCO 
I2.0000CC 
1  3.  0  0  0  0  CO 
14. 0C DC  0  0 
15. 000C0G 
13 . 0  0  0  C  0  0 
21. OCOOOO 
24.  0  0  0  0  C  0 
27. GG00G0 
2G.GOGOGO 
33. 000000 
36.000000 
33 «  0  0  0  0  6  0 


$  x  103 

FCMCAFT 
43;.  .212  1  1  1 
360 . 71n944 
1556.370067 
2557.9 134 4 7 
3^ 06. 09 12- 3 
5002.752145 
7617.70400C 
939^.7 01756 
1 2374.79 0420 
1  49  66 , 49  79  44 
17597. -£3149 
£0  204,0 57  7  1  2 
22726. C95* 09 
25121.997452 
27  361. 77  76  O'1 
330 C7. 242722 
37070.4251 44 
39729.9717 12 
41639. C26263 
42799. 0  76-452 
43532.349215 
43993.072130 
44199. 162492 
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Stephen  Lynn  Amdor  was  born  9  April  1943,  in  Mansfield,  Illinois. 

He  graduated  from  Mansfield  High  School,  Mansfield,  Illinois,  in  1961 
and  then  attended  the  United  States  Air  Force  Academy,  Colorado.  He 
was  graduated  from  that  Institution  in  June,  1965  with  a  Bachelor  of 
Science  Degree  in  Engineering  Management.  Before  attending  Under¬ 
graduate  Pilot  Training  at  Williams  AFB,  Arizona,  he  was  an  assistant 
football  coach  at  the  United  States  Air  Force  Academy  from  June  1965 
to  Mnrch  1966.  Upon  graduation  from  pilot  training,  he  was  assigned 
to  Cannon  AFB,  N.M. ,  for  F-100  combat  crew  training.  Following  that, 
he  served  a  tour  of  combat  in  Southeast  Asia  at  Phu  Cat  AB,  Republic  of 
Vietnam.  He  was  then  assigned  to  RAF  Lakenheath,  England  where  he 
performed  duties  as  a  member  of  the  494th  Tactical  Fighter  Squadron  and 
as  Wing  Standardization/Evaluation  Officer  for  the  48th  Tactical  Fighter 
Wing.  In  July  of  1971,  he  was  assigned  to  Third  Air  Force  Headquarters 
at  South  Ruislip  AS,  England  as  a  Tactical  Fighter  Staff  Officer.  In 
June  1972,  he  entered  the  Air  Force  Institute  of  Technology  as  a 
resident  student  in  Graduate  Systems  Analysis.  He  is  married  to  the 
former  Vallerie  Grace  Petty  and  has  two  children;  Matthew  Todd  and 
Ryan  Frederick.  His  permanent  address  is: 

Box  82,  Mansfield,  Illinois  61854 

Roy  Robert  Kilgore  was  born  4  March  1942,  in  Alamogordo,  New  Mexico. 
He  graduated  from  Alamogordo  High  School,  Alamogordo,  New  Mexico,  in  1960. 
During  the  1960-1961  academic  year  he  attended  New  Mexico  State  University, 
University  Park,  New  Mexico  and  worked  on  a  student  cooperative  project 
in  San  Jose  doe  Campos,  Brazil.  He  entered  the  United  States  Air  Force 
Academy  in  1961  and  was  graduated  from  there  in  1965  with  a  Bachelor  of 
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Science  in  Engineering  Science  Degree.  He  attended  Undergraduate  Pilot 
Training  at  Laredo  AFB,  Texas,  and  was  assigned  from  there  to  MacDill 
AFB,  Florida  for  combat  crew  training  as  a  Pilot  Systems  Operator.  After 
completing  that  training,  he  served  a  combat  tour  at  Cam  Ranh  Bay  AB, 
Republic  of  Vietnam.  In  April  1968.  he  was  assigned  to  George  AFB, 
California  to  upgrade  to  F-4  Aircraft  Commander.  From  there  he  was 
assigned  to  Homestead  AFB,  Florida,  where  he  performed  duties  as  an 
operational  F-4  Aircraft  Commander,  a  Replacement  Training  Unit  F-4 
Instructor  Pilot,  and  a  Wing  Academic  Instructor  for  the  31st  Tactical 
Fighter  Wing.  In  August  1972,  he  entered  the  Air  Force  Institute  of 
Technology  as  a  resident  student  in  Graduate  Systems  Analysis.  He  is 
married  to  the  former  Corina  Ursula  Cantu  and  they  have  two  children: 
Wendy  Ann  and  Matthew  Roy.  His  permanent  address  is: 

Rural  Route  1,  Hotchkiss,  Colorado  81419 
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